DEV Community: Ayooluwa Isaiah

Creating Your First Chrome Extension (With Built-in AI Features)

Ayooluwa Isaiah — Mon, 17 Nov 2025 13:00:00 +0000

Chrome extensions let you customize the browser in ways that go far beyond simple settings or themes. They can block ads, automate repetitive tasks, enhance your favorite sites, or reshape the browser interface entirely.

If you've ever wished a website or the browser itself behaved just a bit differently, chances are an extension already exists to do exactly that.

Learning how to build extensions opens up a space where familiar web technologies gain access to native browser capabilities. You can tap into APIs that ordinary web pages cannot use, allowing your code to run in the background, react to browser events, store data, and integrate deeply into how people work online.

And now, with built-in AI models available directly in Chrome, extensions can go even further. You can generate text on the fly, summarize content, assist users contextually, or build entirely new experiences that adapt to the user's intent.

So whether you're solving a personal frustration or just experimenting with a new distribution channel for your ideas, extensions offer an approachable yet powerful platform. And if you already know HTML, CSS, and JavaScript, you're already 95% of the way there.

In this tutorial, you'll learn the rest of the fundamentals by building a small but fully functional Chrome extension from scratch.

What you'll be building

To understand the various Chrome extensions concepts, you'll build a new tab extension that replaces the default new tab page with a random image from Unsplash.

If you've ever used popular new tab extensions like Tabliss, or Momentum, you already know how transformative this simple UI tweak can be. These extensions turn an empty tab into something visually delightful, functional, or even calming.

Your own version will:

Fetch a random photo from Unsplash.
Cache the image for instant loading each time a new tab opens.
Store user preferences such as their own Unsplash access key and optional photo collections.
Provide a simple settings interface where users can manage these options.
Include a small popup for quick access to the settings page.
Optionally display an AI generated motivational message using Chrome's built-in on-device model.

By the end of this guide, you'll understand how the different parts of an extension fit together: the manifest, service worker, new tab override, options page, popup, and storage.

You'll also see how to use the built-in AI APIs to add a new dimension to what an extension can offer while still keeping everything running locally and securely on the user's device.

Prerequisites

If you have experience building basic web pages, you have everything you need. But make sure you also have:

Google Chrome (the latest version is recommended).
A code editor such as VS Code or any editor you prefer.

Grabbing the starter files

To keep the focus on learning how extensions work rather than writing boilerplate HTML and CSS, this tutorial includes a small set of starter files. They contain the basic layout for the new tab page, the options screen, and the extension's folder structure.

You can download them from GitHub:

git clone https://github.com/freshman-tech/freshtab-starter-files.git

After cloning the repository, move into the project directory:

cd freshtab-starter-files

You can browse the files using your editor or run the tree command (if you have it installed) to get a quick overview of the structure.

tree

├── demo.png
├── icons
│   ├── 128.png
│   ├── 16.png
│   ├── 32.png
│   ├── 48.png
│   └── 64.png
├── index.html
├── js
│   ├── background.js
│   ├── constants.js
│   ├── index.js
│   ├── options.js
│   └── popup.js
├── LICENCE
├── manifest.json
├── options.html
├── package.json
├── popup.html
└── README.md

This gives you a fully prepared workspace so you can focus on the manifest, extension APIs, and the logic that brings everything together.

Anatomy of a Chrome extension

A Chrome extension is simply a collection of files (HTML, CSS, JavaScript, plus a manifest) that work together to add new capabilities to the browser. Before writing any code, it helps to understand the role each file plays in the project.

The manifest

This manifest.json file is the blueprint of your extension. It's a single file that tells Chrome what your extension is, what it does, and the permissions it needs, and other metadata.

The most important thing to know for now is that we are using Manifest V3, the modern standard for all new extensions.

Service worker

A service worker is an event-driven script that wakes up to handle an event (like the extension being installed or a message from another script), does its work, and then terminates.

In this project, the js/background.js file will act as a service worker, and react to browser events and messages to perform various actions in the background.

Popup window

When a user clicks your extension's icon in the toolbar, Chrome can display a small popup. This popup is just a regular HTML page (popup.html) with its own JavaScript and styling. Extensions often use popups to provide quick controls or shortcuts.

Override pages

Extensions can replace certain built in browser pages, such as the new tab page, history page, or bookmarks page. You can override one of these at a time by pointing Chrome to an HTML file in the manifest. In this guide, index.html replaces the default new tab page.

Extension icons

Every extension needs at least one icon so Chrome can display it in the toolbar, extensions page, and Chrome Web Store. The icons used in this tutorial live in the icons folder.

Content scripts

Content scripts are JavaScript files that run directly inside web pages. They can read or modify the page's DOM, making them ideal for features like password managers, ad blockers, and page enhancements.

While this project doesn't use content scripts, they're an important part of many extensions and good to be aware of.

Populating the manifest file

Now that you understand the structure of a Chrome extension, it's time to bring it to life by creating the manifest.json file. This is the central configuration file that Chrome reads first, and without it the browser won't recognize your project as an extension.

Open the manifest.json file in the starter project and update it with the following:

{
  "manifest_version": 3,
  "name": "freshtab",
  "version": "1.0.0",
  "description": "Experience a beautiful photo from Unsplash every time you open a new tab.",
  "icons": {
    "16": "icons/16.png",
    "32": "icons/32.png",
    "48": "icons/48.png",
    "64": "icons/64.png",
    "128": "icons/128.png"
  },
  "chrome_url_overrides": {
    "newtab": "index.html"
  },
  "action": {
    "default_popup": "popup.html"
  },
  "options_page": "options.html",
  "permissions": ["storage"],
  "host_permissions": ["https://api.unsplash.com/"],
  "background": {
    "service_worker": "js/background.js",
    "type": "module"
  }
}

Here's a breakdown of each field in the manifest file:

Required fields

manifest_version: Currently, this must always be 3 as v2 is now deprecated.
name, version: The extension name and version.
description: What the extension does.
icons: Specifies icons for your extension in different sizes.

Optional fields

chrome_url_overrides: Provides a custom replacement for default browser pages (like new tab, history, or bookmarks). In this case, the new tab page is being replaced with the index.html file.
action: Defines the appearance and behavior for the extension's toolbar icon.
options_page: Specifies the path to the options page for your extension.
permissions: Defines the permissions required by the extension. We only need the storage permission here to access the Chrome storage API.
host_permissions: If your extension interacts with a web page or external API, you must list the specific URLs here.
background: Registers the extension's service worker. The type: module line is necessary for the browser to recognize it as an ES module script.

With this manifest in place, Chrome now knows how to load and run your extension. Next, you'll prompt the user for an Unsplash access key so the extension can fetch images.

Prompting users for their Unsplash access key

The core functionality for this extension is fetching and displaying images from Unsplash in each new tab. The following endpoint is provided for this purpose:

https://api.unsplash.com/photos/random

It accepts a number of query parameters to narrow the pool of photos for selection. For our project, we'll use the orientation parameter to limit the results to landscape images alone:

https://api.unsplash.com/photos/random?orientation=landscape

It also requires authentication via the HTTP Authorization header. This is done by setting the Authorization header to Client-ID <ACCESS_KEY> where <ACCESS_KEY> is a valid key from Unsplash.

Follow the instructions on this page to create a free Unsplash account, and register a new application. Once your app is created, copy the Unsplash access key string in the application settings page:

This brings us to a crucial part of building browser extensions responsibly: you must not hardcode your own developer access key into the extension's code. This is insecure (anyone could steal it) and not scalable, as all users would share a single API quota.

The correct, professional approach is to have the user provide their own API key. To make this seamless, you can open the options page automatically when the extension is first installed. This gives users a clear prompt to enter their access key before the extension tries to load any images.

Add the following to your js/background.js file:

chrome.runtime.onInstalled.addListener((details) => {
  if (details.reason === 'install') {
    chrome.runtime.openOptionsPage();
  }
});

This listener runs only when the extension is installed for the first time. Chrome will open the options page in a new tab, letting you add your Unsplash access key right away.

Next, wire up the logic that loads those saved values and stores new ones. Open js/options.js and add:

async function saveOptions() {
  const key = document.getElementById('unsplashKey').value;
  const collections = document.getElementById('collections').value;

  await chrome.storage.local.set({
    unsplashAccessKey: key,
    collections,
  });

  const status = document.getElementById('status');
  status.textContent = 'Options saved.';
  setTimeout(() => {
    status.textContent = '';
  }, 1500);
}

async function restoreOptions() {
  const result = await chrome.storage.local.get([
    'unsplashAccessKey',
    'collections',
  ]);

  document.getElementById('unsplashKey').value = result.unsplashAccessKey || '';
  document.getElementById('collections').value = result.collections || '';
}

document.addEventListener('DOMContentLoaded', restoreOptions);
document.getElementById('save').addEventListener('click', saveOptions);

This script does two things:

Loads saved settings whenever the options page opens.
Saves new settings to chrome.storage.local when users click the Save Settings button.

We're using chrome.storage.local to save the API key and collection IDs to the local machine only. If you need to sync certain extension settings across all Chrome browsers where the user is logged in, you may use the chrome.storage.sync API instead.

The trade-off is a much smaller storage quota (around 100 KB), but that's usually enough for most settings objects.

Once these pieces are in place, load your extension in Chrome by typing chrome://extensions in the address bar and select Load unpacked after enabling Developer mode:

Navigate to the root of your project directory (where the manifest.json file lives) and select it. When the extension installs, the options page will open automatically and you can enter your Unsplash key.

You can confirm this by opening the Chrome DevTools with F12, then go to the Application tab and find the Extension storage > Local entry. You should see the empty collections field and the unsplashAccessKey key that you just added:

With the access key safely stored, the extension is ready to start fetching images in the background.

When you open a new browser tab, it should already be replaced by the one defined in your extension manifest (index.html). This page is currently blank as shown below:

You may need to turn off the Footer in recent versions of Chrome. Click Customize Chrome or go to More tools > Customize Chrome in the browser menu to fond the relevant option:

Fetching images from Unsplash

With the settings page wired up and your new tab page ready to render content, the next step is to actually load an image. That work lives in the service worker, which will:

Read the saved API key and optional collection IDs from chrome.storage.
Call the Unsplash API to fetch a random photo.
Store the image response in the browser cache using the\ Cache API.

Update your js/background.js file to look like this:

// js/background.js
import { CACHE_NAME, IMAGE_KEY, METADATA_KEY } from './constants.js';

chrome.runtime.onInstalled.addListener((details) => {
  if (details.reason === 'install') {
    chrome.runtime.openOptionsPage();
  }

  // Always fetch a new image on install or update
  fetchAndCacheImage();
});

async function fetchAndCacheImage() {
  try {
    const { unsplashAccessKey } = await chrome.storage.local.get(
      'unsplashAccessKey'
    );
    if (!unsplashAccessKey) {
      chrome.runtime.openOptionsPage();
      return;
    }

    const metaResponse = await fetchPhotoMetadata(unsplashAccessKey);
    const imageResponse = await fetch(metaResponse.urls.raw + '&q=85&w=2000');
    if (!imageResponse.ok) {
      throw new Error('Failed to fetch the image file.');
    }

    const cache = await caches.open(CACHE_NAME);
    await cache.put(IMAGE_KEY, imageResponse);

    await chrome.storage.local.set({ [METADATA_KEY]: metaResponse });
  } catch (err) {
    console.error('Error fetching and caching image:', err);
  }
}

async function fetchPhotoMetadata(apiKey) {
  const { collections } = await chrome.storage.local.get('collections');

  let endpoint = 'https://api.unsplash.com/photos/random?orientation=landscape';
  if (collections) {
    endpoint += `&collections=${collections}`;
  }

  const headers = new Headers();
  headers.append('Authorization', `Client-ID ${apiKey}`);

  const response = await fetch(endpoint, { headers });
  if (!response.ok) {
    throw new Error(`Unsplash API error: ${response.statusText}`);
  }

  return response.json();
}

The script uses the chrome.runtime.onInstalled listener to detect when a user first installs or updates the extension. On a fresh install it opens the options page, and in all cases it calls fetchAndCacheImage() so there is at least one image ready.

The fetchAndCacheImage() function checks chrome.storage.local for the user's saved unsplashAccessKey. If the key is missing, it simply re-opens the options page and stops.

If the key is present, it calls the fetchPhotoMetadata() helper to get the photo's data (like its download URL and EXIF info). This helper builds the correct Unsplash API URL, attaches the API key to the Authorization header, and adds any custom collection IDs (if configured).

Once it has the metadata, fetchAndCacheImage() makes a second fetch call to get the actual image file and requests a resized version for better performance. It then opens the cache and uses cache.put to store the image data.

Finally, it saves the metadata object into chrome.storage.local, where all extension scripts can easily access it.

The constants referenced above are defined in js/constants.js:

// js/constants.js
export const CACHE_NAME = 'freshtab-image-cache-v1';
export const IMAGE_KEY = 'https://freshtab-app.local/next-image';
export const METADATA_KEY = 'next-image';

Why we're using the Cache API

This extension needs to store two different kinds of data:

A small JSON object with metadata.
A binary image file for the new tab background.

chrome.storage.local is designed for JSON serializable values like strings, numbers, or objects. It cannot store large binary data like an image Response or a Blob.

That's why the Cache API is used for the image instead. However, note that the IMAGE_KEY does not need to point to a real website. It only needs to be a valid https:// URL so the cache accepts it (otherwise you'll get an error).

If you want to store the image binary in chrome.storage.local instead, you would need to convert it to a Base64 string and add the unlimitedStorage permission, since high resolution images can easily exceed the default 10 MB quota.

Testing it out

You can now reload the extension and confirm everything is working. Find the freshtab entry in chrome://extensions and click the reload button. After a few moments, open a new tab and launch the Chrome DevTools.

In the DevTools Application panel under Extension storage → Local, you should see a next-image entry containing the metadata for the next tab's background image:

Under Cache storage, open freshtab-image-cache-v1 to see the saved image response stored under the IMAGE_KEY URL:

With the image successfully cached, you are ready to display it on the new tab page.

Loading the Unsplash image on every new tab

Now that the Unsplash metadata and image are being saved to the extension's local storage and cache, the new tab page can read them and display the photo.

The script responsible for this is js/index.js which runs every time a new tab opens. Its job is to retrieve the cached image, show it as quickly as possible, and notify the service worker to prepare the next image.

Open your js/index.js file and add the following code:

// js/index.js
import { CACHE_NAME, IMAGE_KEY } from './constants.js';

document.addEventListener('DOMContentLoaded', () => {
  main();
});

async function main() {
  await loadPhoto();

  chrome.runtime.sendMessage({ command: 'next-image' });
}

async function loadPhoto() {
  const cache = await caches.open(CACHE_NAME);
  const cachedResponse = await cache.match(IMAGE_KEY);

  if (cachedResponse) {
    const blob = await cachedResponse.blob();
    document.body.style.backgroundImage = `url(${URL.createObjectURL(blob)})`;
  } else {
    document.body.style.backgroundColor = '#111';
  }
}

When the page's HTML has loaded, the main() function is triggered. This function does two key things:

It immediately calls loadPhoto() to set the background image.
It then sends a next-image message to the service worker, telling it to fetch a new photo in the background so it is ready for the next tab.

The loadPhoto() function opens the browser cache and looks for an entry that matches IMAGE_KEY. If it finds one, it converts the cached response into a blob URL and applies it as the page background.

This way, a network request is avoided, making the image appear almost instantly. If nothing is cached yet (for example, on the first run), it falls back to the previous dark background color.

Before testing this, update your service worker so it listens for the next-image command:

// js/background.js
// [...]

// Listen for the "next-image" command from the new tab page
chrome.runtime.onMessage.addListener((request) => {
  if (request.command === 'next-image') {
    fetchAndCacheImage();
  }
});

Now reload the extension from the chrome://extensions page. Open a new tab and you should be greeted with an image from Unsplash:

As you continue opening new tabs, you should see a different image each time, thanks to the next-image message the page sends and the service worker handles.

Creating a simple settings popup

To make the extension's settings easier to reach, let's add a small popup that opens whenever the user clicks the extension's toolbar icon.

The popup won't do much, but it gives users an obvious way to access the options page without digging through menus.

Here is the popup.html file:

<!DOCTYPE html>
<html>
  <head>
    <title>Freshtab</title>
    <link
      rel="stylesheet"
      href="https://cdn.jsdelivr.net/npm/bulma@1.0.0/css/bulma.min.css"
    />
    <style>
      body {
        width: 200px;
        padding: 1.25rem;
      }
    </style>
  </head>
  <body>
    <h3 class="title is-5 has-text-centered">Freshtab</h3>

    <button class="button is-primary is-fullwidth" id="options-btn">
      Go to Settings
    </button>

    <script src="js/popup.js"></script>
  </body>
</html>

The popup's JavaScript is simple. It listens for a click on the button and opens the extension's main options page:

// js/popup.js
document.addEventListener('DOMContentLoaded', () => {
  const optionsButton = document.getElementById('options-btn');

  if (optionsButton) {
    optionsButton.addEventListener('click', () => {
      chrome.runtime.openOptionsPage();
    });
  }
});

When you click the extension icon in Chrome's toolbar, you'll see a compact window with a single button:

While users can also right-click the icon and choose Options, many people don't know that menu exists. A dedicated popup makes the settings far more discoverable and feels more polished.

If you like, you can take this further by adding a settings button directly on the new tab page as well.

Restricting images to user-defined collections

By default, the extension selects a random landscape photo from the entire Unsplash library. To make the experience more personal, you can let users provide one or more Unsplash Collection IDs. When set, the extension will pull images only from those specific collections.

This feature is already built into both the options page and the fetchPhotoMetadata() function in your background.js script, so no additional code is required.

To try it out, visit the Unsplash collections page and open any collection. The collection ID appears in the page URL, as shown below:

Copy that ID, paste it into the Custom Collection IDs field on the options page, and click Save Settings:

From now on, whenever the extension fetches an image, it will restrict the request to the specified collection. The result is a curated, more consistent set of images:

Adding motivational quotes with Chrome's Built-in AI

One of the most interesting changes in recent versions of Chrome is the addition of a built in language model that runs directly on the user's device. Through the LanguageModel API, your extension can generate text without sending data to a remote server.

But there are a few caveats.

First, this is not a universal feature; it comes with strict hardware and software requirements. The on-device AI model, Gemini Nano, requires a desktop machine with at least 16 GB of RAM if running on the CPU, or a GPU with more than 4 GB of VRAM.

It's also not available out of the box. The user must have an unmetered internet connection and at least 22 GB of free disk space for the initial model download.

And you can't just start this download automatically even if the baseline requirements are met; you have to wait for a user activation (like a button click) before you can trigger the download by creating a session.

Because of all this, you can't assume the feature will work for everyone, so it's essential to build in a graceful fallback. For this tutorial, this means checking if the model is available and simply not showing the quote if it isn't.

Let's begin by exposing a setting that lets users turn this feature on or off. In your options.html file, find and uncomment the following block:

<!-- options.html -->
<div class="field">
  <label class="label">AI-powered motivational message</label>
  <div class="control">
    <label class="checkbox" id="quote-label">
      <input type="checkbox" id="showQuote" />
      Show a motivational quote on the new tab page from Chrome's built-in AI
      model (Gemini Nano).
    </label>
  </div>
  <p class="help" id="quote-help">
    When enabled, a new quote will be fetched for each tab.
  </p>
</div>

Since this feature depends on a browser API that may not be available everywhere, the options page needs to detect support first and update the UI accordingly.

Open js/options.js and replace its contents as follows:

// js/options.js
async function saveOptions() {
  const key = document.getElementById('unsplashKey').value;
  const collections = document.getElementById('collections').value;
  const showQuote = document.getElementById('showQuote').checked;

  await chrome.storage.local.set({
    unsplashAccessKey: key,
    collections,
    showQuote,
  });

  const status = document.getElementById('status');
  status.textContent = 'Options saved.';
  setTimeout(() => {
    status.textContent = '';
  }, 1500);
}

async function restoreOptions() {
  const result = await chrome.storage.local.get([
    'unsplashAccessKey',
    'collections',
    'showQuote',
  ]);

  document.getElementById('unsplashKey').value = result.unsplashAccessKey || '';
  document.getElementById('collections').value = result.collections || '';
  document.getElementById('showQuote').checked = result.showQuote ?? false;
}

async function checkAIAvailability() {
  const quoteCheckbox = document.getElementById('showQuote');
  const quoteLabel = document.getElementById('quote-label');
  const quoteHelp = document.getElementById('quote-help');

  const markUnavailable = () => {
    quoteCheckbox.disabled = true;
    quoteLabel.classList.add('is-disabled');

    quoteHelp.textContent = 'This feature is not available in your browser';
    quoteHelp.classList.add('is-danger');
  };

  if (!('LanguageModel' in self)) {
    markUnavailable();
  }

  const availability = await LanguageModel.availability();

  if (availability === 'available') return;

  if (availability === 'unavailable') {
    return markUnavailable();
  }
}

document.addEventListener('DOMContentLoaded', async () => {
  restoreOptions();

  await checkAIAvailability();
});
document.getElementById('save').addEventListener('click', saveOptions);

Here's what this code does:

restoreOptions() loads any saved settings, including showQuote.
saveOptions() persists the Unsplash key, collections, and the AI quote toggle.
checkAIAvailability() calls LanguageModel.availability() and, if the feature is missing, disables the checkbox and shows a helpful message.

If the API is available, the checkbox remains enabled and users can turn the feature on as normal.

You also need to handle the case where the model is downloadable but not yet installed. In that case, you need to trigger the download (which happens in the background) and surface progress so that users know what's happening.

To implement this, extend js/options.js with:

async function saveOptions() {
  // [...]
  if (showQuote) {
    createModelSession();
  }
}

async function createModelSession() {
  const progress = document.getElementById('model-progress');
  const quoteHelp = document.getElementById('quote-help');

  try {
    quoteHelp.textContent = 'Initializing download...';
    progress.classList.remove('is-hidden');

    const availability = await LanguageModel.availability();
    if (availability === 'available') return;

    const session = await LanguageModel.create({
      monitor(m) {
        m.addEventListener('downloadprogress', (e) => {
          quoteHelp.textContent = `Downloading AI model... (${Math.round(
            e.loaded * 100
          )}%)`;
          progress.value = e.loaded;

          if (e.loaded === 1) {
            quoteHelp.textContent = 'Download complete, model installed';
            progress.removeAttribute('value');
          }
        });
      },
    });

    session.destroy();
  } catch (error) {
    quoteHelp.textContent = error.message;
    quoteHelp.classList.add('is-danger');
    console.log(error);
  } finally {
    progress.classList.add('is-hidden');
    progress.value = 0;
  }
}

When the user enables the AI feature and clicks Save Settings, createModelSession() is executed. It:

Displays a progress bar.
Calls LanguageModel.create() with a monitor to track download progress.
Updates the help text as the model downloads and installs.

Once the download and installation is completed, the model will become accessible for prompting.

Next, wire the LanguageModel API into the new tab script itself. Open js/index.js and update it as follows:

// js/index.js
import { CACHE_NAME, IMAGE_KEY } from './constants.js';

document.addEventListener('DOMContentLoaded', () => {
  main();
  displayQuote();
});

// [...]
async function runPrompt(prompt, params) {
  let session;

  try {
    if (!session) {
      session = await LanguageModel.create(params);
    }
    return session.prompt(prompt);
  } catch (e) {
    console.log('Prompt failed');
    console.error(e);
    console.log('Prompt:', prompt);
    session.destroy();
  }
}

async function displayQuote() {
  const { showQuote } = await chrome.storage.local.get('showQuote');
  if (!showQuote) return;

  if (!('LanguageModel' in self)) return;

  const prompt =
    'Write a one-sentence motivational message about success, perseverance or discipline';

  const params = {
    expectedInputs: [{ type: 'text', languages: ['en'] }],
    expectedOutputs: [{ type: 'text', languages: ['en'] }],
    temperature: 2,
    topK: 128,
  };
  const availability = await LanguageModel.availability(params);

  if (availability !== 'available') {
    return;
  }

  const quoteText = document.getElementById('quote-text');
  const quoteAuthor = document.getElementById('quote-author');

  try {
    quoteText.textContent = 'Loading quote...';
    const response = await runPrompt(prompt, params);
    quoteText.textContent = response;
    quoteAuthor.classList.remove('is-hidden');
  } catch (e) {
    console.log(e);
    quoteText.textContent = '';
  }
}

This script is what brings the new AI quote feature to life on your new tab page.

The displayQuote() function checks chrome.storage.local to see if the feature is enabled. If so, it checks that the LanguageModel API exists and that the model is available with the requested parameters. It then sets a simple prompt asking for a short motivational sentence.

The expectedInputs and expectedOutputs fields specify that you want plain text in English, while temperature and topK are set to high values to encourage more varied output. For more ideas and tuning tips, check out the Prompt API documentation and prompting strategies guide

While the prompt is running, the UI shows a Loading quote... message so the user knows something is happening. When a response arrives, the quote text replaces that message and the quote author element is revealed.

After refreshing the new tab page, you should start seeing AI generated motivational messages from Gemini Nano running locally in your browser!

You can experiment with different prompts and parameters to change the tone, length, or style of the messages.

Just keep in mind that on-device AI models are not designed to handle the same level of complexity as the massive server-side models running in the cloud.

Some tips for debugging Chrome extensions

Extensions are made up of several moving parts: a service worker, popup, options page, and any overridden pages. When something breaks you need to know which piece to inspect.

Your main tool is the chrome://extensions page. Just make sure Developer mode is toggled on in the top-right corner.

To debug your service worker (background.js), find your extension card on the extensions page and click the service worker link:

Chrome opens a dedicated DevTools window where you can:

View console.log output from the service worker.
Step through code in the Sources panel.
Inspect network calls, including Unsplash API requests.
Inspect extension storage.

If your Chrome extension encounters a serious problem, you may also see an Errors button on the extension card:

Click it to open a panel that shows recent errors and stack traces:

After fixing the underlying issue, you can clear the errors and reload the extension.

To debug the popup, right click the extension icon in the toolbar and choose Inspect popup. This opens a DevTools window attached to the popup and keeps the popup from closing while you inspect it:

Finally, for new tab pages (or other browser overrides) you can debug them like a regular web page by right-click anywhere on the page and selecting Inspect or using the F12 shortcut key.

Resources and next steps

You've now built a full Chrome extension from scratch and gained a solid understanding of how extension parts fit together, how Manifest V3 works, and how to interact with the browser through its extension APIs.

You can see the full code in the final branch on GitHub

As you take on more advanced ideas, here are some helpful directions to explore:

Using an extensions framework

For larger or more complex projects, writing everything by hand can get cumbersome. Frameworks like Wxt and Plasmo handle the boring stuff for you such as automatic reloading, streamlining builds, and making it easier to integrate modern JavaScript tooling.

The official Chrome Extensions Samples repository is also a great place to see real, working examples of every major API.

Going deeper with Chrome's AI features

If you want to continue experimenting with the LanguageModel API, Chrome's documentation offers practical examples and prompt patterns.

With these tools and references, you can expand this project into something more interesting or use it as a launching point for entirely new ideas.

Once you're happy with your extension, you can publish it to the Chrome Web Store so others can install it. Google provides a step-by-step guide for the full submission process.

Happy building!

A Complete Guide to Node.js Process Management with PM2

Ayooluwa Isaiah — Wed, 23 Mar 2022 12:31:02 +0000

Process management refers to various activities around the creation, termination, and monitoring of processes. A process manager is a program that ensures that your applications always stay online after being launched.

Process managers can prevent downtime in production by automatically restarting your application after a crash or even after the host machine reboots. They are also useful in development: they auto-restart an app once its source files or dependencies are updated. Process managers also typically provide monitoring tools that access application logs and other key metrics, such as CPU and memory usage.

PM2 is a Node.js process manager that comes with a built-in load balancer. It helps facilitate production deployments and enables you to keep running applications alive indefinitely (even when accidents occur). It also lets you gain insights into your application's runtime performance and resource consumption and scale your application in real-time through its clustering feature.

In this article, we'll examine PM2's most important features and discover how it can help you increase the resilience of your Node.js applications in production.

Getting Started with PM2

PM2 is available as an NPM package, so you can install it through npm or yarn:

$ npm install pm2
# or
$ yarn add pm2

After installing PM2, run npx pm2 --version to see the installed version:

$ npx pm2 --version
5.1.2

If you don't want to prefix the pm2 command with npm every time, you can install it globally:

$ npm install -g pm2
# or
$ yarn global add pm2

Aside from the main pm2 command, the installation provides some other executables:

pm2-dev: a development tool for restarting your application when file changes in the directory are detected (similar to Nodemon).
pm2-runtime: designed to be a drop-in replacement for the node binary in Docker containers. It helps keep the running application in the foreground (unlike pm2, which sends it to the background) so that the container keeps running.
pm2-docker: an alias for pm2-runtime.

Start Your Node.js App in Development Mode with PM2

It can be quite tedious to restart your application server in development every time you change the source files. Using the pm2-dev binary to start your application can take care of that concern automatically:

$ pm2-dev start app.js
===============================================================================
--- PM2 development mode ------------------------------------------------------
Apps started         : app
Processes started    : 1
Watch and Restart    : Enabled
Ignored folder       : node_modules
===============================================================================
app-0  | {"level":30,"time":1638512528047,"pid":4575,"hostname":"Kreig","msg":"Server listening at http://127.0.0.1:3000"}
[rundev] App app restarted
app-0  | {"level":30,"time":1638512535737,"pid":4631,"hostname":"Kreig","msg":"Server listening at http://127.0.0.1:3000"}

At this point, your server will auto-restart each time you create, modify or delete a source file in your project. It also works when you add or remove a dependency with npm or yarn.

Start Your Node.js App in Production Mode with PM2

When deploying an application to production, you can use the pm2 binary to start it in the background. It launches a daemon that monitors your application and keeps it running indefinitely.

$ pm2 start app.js
[PM2] Starting /home/ayo/dev/demo/covid-node/app.js in fork_mode (1 instance)
[PM2] Done.
┌─────┬────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id  │ name   │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├─────┼────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0   │ app    │ default     │ 1.0.0   │ fork    │ 16573    │ 0s     │ 0    │ online    │ 0%       │ 19.1mb   │ ayo      │ disabled │
└─────┴────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘

PM2 defaults to the name of the entry file as the app's name, but you can use a more recognizable name through the --name option. This name is what you'll use to reference the application in many pm2 subcommands.

$ pm2 start app.js --name "my app"

Suppose you need to ensure that your application has established connections with other services (such as the database or cache) before being considered "online" by PM2. In that case, you can use the --wait-ready option when starting your application. This causes PM2 to wait for 3 seconds (by default) or for a ready event (process.send('ready')) before the application is considered ready. You can use the --listen-timeout option to change the length of the delay.

$ pm2 start app.js --wait-ready --listen-timeout 5000 # wait for 5 seconds

Monitoring Your Running Applications in PM2

To list your running applications, use the pm2 list command. This prints a table describing the state of all running applications with columns for:

the app name and id
CPU and memory usage
number of restarts (↺)
uptime
process id
the mode (fork or cluster)

and others.

You can use this table alongside a host monitoring service like AppSignal to give you a complete picture of your application and its host environment:

$ pm2 list
┌─────┬───────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id  │ name      │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├─────┼───────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0   │ app       │ default     │ 1.0.0   │ fork    │ 16573    │ 9m     │ 0    │ online    │ 0%       │ 57.3mb   │ ayo      │ disabled │
│ 2   │ index     │ default     │ 1.0.0   │ fork    │ 0        │ 0      │ 16   │ errored   │ 0%       │ 0b       │ ayo      │ disabled │
│ 1   │ server    │ default     │ 0.1.0   │ fork    │ 17471    │ 71s    │ 0    │ online    │ 0%       │ 77.5mb   │ ayo      │ disabled │
└─────┴───────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘

If you see only a subset of this information, try enlarging your terminal window. The list subcommand will not display all the columns if your terminal window is too small. You can also sort the output table according to a metric of your choice:

$ pm2 list --sort [name|id|pid|memory|cpu|status|uptime][:asc|desc]
# such as
$ pm2 list --sort uptime:desc

If you require more information about a particular app beyond what list provides, use the show subcommand and pass the app name to view more detailed application process metadata. Some of the metrics and data presented in the output include the app's:

output and error log files
heap size and usage
event loop latency
uptime
number of restarts
source control metadata

and more.

$ pm2 show server
Describing process with id 1 - name server
┌───────────────────┬──────────────────────────────────────────────────┐
│ status            │ online                                           │
│ name              │ server                                           │
│ namespace         │ default                                          │
│ version           │ 0.1.0                                            │
│ restarts          │ 0                                                │
│ uptime            │ 60m                                              │
│ script path       │ /home/ayo/dev/demo/analytics-dashboard/server.js │
│ script args       │ N/A                                              │
│ error log path    │ /home/ayo/.pm2/logs/server-error.log             │
│ out log path      │ /home/ayo/.pm2/logs/server-out.log               │
│ pid path          │ /home/ayo/.pm2/pids/server-1.pid                 │
│ interpreter       │ node                                             │
│ interpreter args  │ N/A                                              │
│ script id         │ 1                                                │
│ exec cwd          │ /home/ayo/dev/demo/analytics-dashboard           │
│ exec mode         │ fork_mode                                        │
│ node.js version   │ 17.0.0                                           │
│ node env          │ N/A                                              │
│ watch & reload    │ ✘                                                │
│ unstable restarts │ 0                                                │
│ created at        │ 2021-12-03T08:33:01.489Z                         │
└───────────────────┴──────────────────────────────────────────────────┘

. . .

Another way to keep tabs on your running applications is through the built-in terminal dashboard (accessed through the monit subcommand). This allows you to view live data on resource usage and logs for each of your applications.

$ pm2 monit

Restarting Your Node.js Application with PM2

PM2 allows you to configure several different strategies for how your Node.js application should restart. By default, it restarts your application if it exits or crashes to minimize the impact to your customers in production while the source of the crash is investigated. The restart subcommand is also available for manually restarting your application at any time.

$ pm2 restart app

To ensure a graceful shutdown, make sure you intercept the SIGINT signal to stop all new requests and finish up existing ones before allowing your program to exit.

process.on('SIGINT', function() {
   gracefulShutdown((err) => {
     process.exit(err ? 1 : 0)
   });
})

You can use the --kill-timeout option to ensure that a graceful shutdown does not take too long:

$ pm2 restart app --kill-timeout 5000 # set a 5 second limit

Auto Restart Based on Memory Usage

The --max-memory-restart option is available to restart an app when it reaches a certain memory threshold. This can help prevent a Node.js heap out of memory error. You can specify the memory limit in kilobytes
(K), Megabytes (M), or Gigabytes (G).

$ pm2 start app.js --max-memory-restart 1G

Auto Restart Based on Cron Schedule

PM2 also offers a restart strategy based on the Cron
syntax.

This allows you to schedule a restart at a specific time each day / on certain days of the week / a set time interval (such as every 48 hours).

# Restart at 12:00 pm every day
$ pm2 start app.js --cron-restart="0 12 * * *"

Auto Restart on File Change

Remember how pm2-dev auto-restarts your application when you make changes to a file? You can configure the pm2 command to act in a similar manner through the --watch subcommand. In the table outputted by pm2 list, look at the watching column to observe the watch status of an application.

$ pm2 start app.js --watch

Auto Restart after a Delay

You can configure the --restart-delay option to set a delay for automatic restarts. The delay should be supplied in milliseconds.

$ pm2 start app.js --restart-delay=5000 # 5s delay

Ignore Some Exit Codes When Auto Restarting

PM2 auto-restarts your app when the process exits, but it does not take the exit code into account by default, so it restarts regardless of whether the app exits cleanly or crashes. If this behavior is not desired, you can use the --stop-exit-codes option to set exit codes that should not prompt PM2 to auto-restart. For example, you can ensure PM2 does not auto-restart on a clean exit with the following command:

$ pm2 start app.js --stop-exit-codes 0

Restarting Processes after a System Reboot

The previous section covered a variety of ways to restart your application after it is launched. However, none of the strategies there will keep your application up if your server reboots. Notably, PM2 ships with a startup feature that can help solve this problem. You can combine this with a good uptime monitoring service like AppSignal's to guarantee that your application comes back online quickly, even if an accident happens.

You'll need to generate a startup script for your server's init system to execute on system boot and launch the PM2 process, which will subsequently start the configured application processes immediately. You can allow PM2 to autodetect your startup script or pass the init system used by your operating system, which could be systemd, upstart, launchd, rcd, or systemv.

$ pm2 startup # autodetect init system
# or
$ pm2 startup systemd # generate script for systemd

You should receive the following output:

[PM2] Init System found: systemd
-----------------------------------------------------------
 PM2 detected systemd but you precised systemd
 Please verify that your choice is indeed your init system
 If you arent sure, just run : pm2 startup
-----------------------------------------------------------
[PM2] To setup the Startup Script, copy/paste the following command:
sudo env PATH=$PATH:/usr/bin /usr/local/lib/node_modules/pm2/bin/pm2 startup systemd -u ayo --hp /home/ayo

You'll need to copy and paste the generated command into the terminal, and then run it as the root:

$ sudo env PATH=$PATH:/usr/bin /usr/local/lib/node_modules/pm2/bin/pm2 startup <distribution> -u <user> --hp <home-path>

If everything goes well, you'll see the following output, indicating that PM2 is configured to start at boot.

[PM2] Init System found: systemd

. . .

[PM2] [v] Command successfully executed.
+---------------------------------------+
[PM2] Freeze a process list on reboot via:
$ pm2 save

[PM2] Remove init script via:
$ pm2 unstartup systemd

At this point, you can run pm2 save to save your process list. This saves the processes currently managed by PM2 to disk so they're accessible to the daemon on system boot.

$ pm2 save
[PM2] Saving current process list...
[PM2] Successfully saved in /home/<user>/.pm2/dump.pm2

Go ahead and restart your computer or server. Once it boots back up, run pm2 list to see if all the processes are restored. If PM2 doesn't restore them automatically, you can manually relaunch them with the resurrect subcommand. You then won't need to start each process individually.

$ pm2 resurrect
[PM2] Resurrecting
[PM2] Restoring processes located in /home/<user>/.pm2/dump.pm2

At any point in the future, you can run pm2 save again to update the list of processes that should be restored on boot or when using the resurrect subcommand.

Clustering with PM2

Clustering in Node.js refers to creating child processes that run simultaneously and share the same port in an application. This technique makes it possible to horizontally scale a Node.js application on a single machine, taking advantage of the processing capabilities offered by multi-core systems (since an instance of a Node.js app only runs on a single thread).

The standard Node.js library provides a cluster module to set up clustering in Node.js applications. In a nutshell, it creates child processes (workers) and distributes incoming connections across the simultaneously running worker processes. You'll need to modify your source code to spawn and manage the workers and set up how you'd like to distribute incoming connections amongst them.

PM2 also provides a cluster mode that uses the native cluster module under the hood. However, it does not require any modifications to the application's source code. Instead, all you need to do to start a Node.js program in cluster mode is to supply the -i option to the start subcommand, as follows:

$ pm2 start app.js -i 0
┌────┬────────────────────┬──────────┬──────┬───────────┬──────────┬──────────┐
│ id │ name               │ mode     │ ↺    │ status    │ cpu      │ memory   │
├────┼────────────────────┼──────────┼──────┼───────────┼──────────┼──────────┤
│ 0  │ app                │ cluster  │ 0    │ online    │ 0%       │ 49.0mb   │
│ 1  │ app                │ cluster  │ 0    │ online    │ 0%       │ 46.8mb   │
│ 2  │ app                │ cluster  │ 0    │ online    │ 0%       │ 44.8mb   │
│ 3  │ app                │ cluster  │ 0    │ online    │ 0%       │ 42.2mb   │
└────┴────────────────────┴──────────┴──────┴───────────┴──────────┴──────────┘

The -i or instances option above allows you to specify the number of workers (child processes) that PM2 should launch. You can set 0 or max to specify that PM2 should spawn as many workers as the number of available CPU cores (as above). Alternatively, you can set the exact number of workers to be greater than the number of available CPU cores, if desired. If you want to add additional worker processes on the fly, use the scale subcommand as shown below:

$ pm2 scale <app_name> +4 # add 4 additional workers in realtime

Once your application launches in cluster mode, incoming requests to the server will be automatically load-balanced across all the worker processes, which can significantly improve throughput. This feature also enables you to restart your app in production (using pm2 restart) without suffering any downtime since PM2 waits for the new workers to become operational before it kills the old ones.

PM2's clustering feature works best when your application is completely stateless. You won't need any code modifications to scale on the same server or even across multiple servers if your app doesn't maintain any state in individual processes. If your application isn't stateless, you'll likely get better results directly using the native cluster module.

Log Management in PM2

Log management is quite straightforward in PM2. The logs for all your running applications are placed in the ~/.pm2/logs directory, and they can be displayed with the logs subcommand. All log entries are prefixed with the application's name to ensure easy identification.

$ pm2 logs # display all logs in realtime
$ pm2 logs <app_name> # display only a specific app's logs

You can also clear log data with the flush subcommand:

$ pm2 flush # clear all log data
$ pm2 flush <app_name> # flush log data for a specific app

To enable log rotation, install the following module:

$ pm2 install pm2-logrotate

Wrap Up and Next Steps: Dive Further into PM2

I hope this article has helped to crystallize the importance of process management in Node.js applications and how to leverage PM2's robust feature set to manage your application efficiently.

PM2 offers other capabilities that were not covered in this article such as Docker integration, a JavaScript API, and a daemon-less mode, so ensure you check out PM2's documentation to learn more about these advanced features.

Thanks for reading, and happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

An Introduction to Deno: Is It Better than Node.js?

Ayooluwa Isaiah — Wed, 16 Feb 2022 12:40:58 +0000

Deno is a JavaScript and TypeScript runtime similar to Node.js, built on Rust and the V8 JavaScript engine. It was created by Ryan Dahl, the original inventor of Node.js, to counter mistakes he made when he originally designed and released Node.js back in 2009.

Ryan's regrets about Node.js are well documented in his famous '10 Things I Regret About Node.js' talk at JSConf EU in 2018. To summarize, he bemoaned the lack of attention to security, module resolution through node_modules, various deviations from how browser's worked, amongst other things, and he set out to fix all these mistakes in Deno.

In this article, we'll discuss why Deno was created and its advantages and disadvantages compared to Node.js. It'll also give a practical overview of Deno's quirks and features so that you can decide if it's a good fit for your next project.

Installing Deno

Deno is distributed as a single, self-contained binary without any dependencies.

You can install Deno in various ways, depending on your operating system. The simplest method involves downloading and executing a shell script as shown below:

# Linux and macOS
$ curl -fsSL https://deno.land/x/install/install.sh | sh

# Windows PowerShell
$ iwr https://deno.land/x/install/install.ps1 -useb | iex

Once you've executed the appropriate command for your operating system, the Deno CLI binary will be downloaded to your computer. You may be required to add the binary location to your PATH, depending on the installation method you chose.

You can do this in Bash by adding the lines below to your
$HOME/bash_profile file. You may need to start a new shell session for the changes to take effect.

export DENO_INSTALL="$HOME/.deno"
export PATH="$DENO_INSTALL/bin:$PATH"

To verify the installed version of Deno, run the command below. It should print the Deno version to the console if the CLI was downloaded successfully and added to your PATH.

$ deno --version
deno 1.14.2 (release, x86_64-unknown-linux-gnu)
v8 9.4.146.16
typescript 4.4.2

If you have an outdated version of Deno, upgrading to the latest release can be done through the upgrade subcommand:

$ deno upgrade
Looking up latest version
Found latest version 1.14.2
Checking https://github.com/denoland/deno/releases/download/v1.14.2/deno-x86_64-unknown-linux-gnu.zip
31.5 MiB / 31.5 MiB (100.0%)
Deno is upgrading to version 1.14.2
Archive:  /tmp/.tmpfdtMXE/deno.zip
  inflating: deno
Upgraded successfully

Go ahead and write a customary hello world program to verify that everything works correctly. You can create a directory for your Deno programs and place the following code in an index.ts file at the directory's root.

function hello(str: string) {
  return `Hello ${str}!`;
}

console.log(hello("Deno"));

Save and execute the file by providing the filename as an argument to the run subcommand. If the text "Hello Deno!" outputs, it means that you have installed and set up Deno correctly.

$ deno run index.ts
Check file:///home/ayo/dev/deno/index.ts
Hello Deno!

To find out about other features and options provided by the Deno CLI, use the --help flag:

$ deno --help

Deno's First-class TypeScript Support

One of the big selling points of Deno over Node.js is its first-class support for TypeScript.

As you've already seen, you don't need to do anything besides install the Deno CLI for it to work. Like its predecessor, Deno uses the V8 runtime engine under the hood to parse and execute JavaScript code, but it also includes the TypeScript compiler in its executable to achieve TypeScript support.

Under the hood, TypeScript code is checked and compiled. The resulting JavaScript code is cached in a directory on your filesystem, ready to be executed again without being compiled from scratch. You can use deno info to inspect the location of the cache directory and other directories containing Deno-managed files.

Deno does not require any configuration to work with TypeScript, but you can provide one if you want to tweak how the TypeScript compiler parses the code. You can provide a JSON file to specify the TypeScript compiler options. Although tsconfig.json is the convention when using the standalone tsc compiler, the Deno team recommends using deno.json because other Deno-specific configuration options can be placed there.

Note that Deno doesn't support all TypeScript compiler options. A full list of the available options, along with their default values, are presented in the Deno documentation. Here's a sample configuration file for Deno:

{
  "compilerOptions": {
    "checkJs": true,
    "noImplicitReturns": true,
    "noUnusedLocals": true,
    "noUnusedParameters": true,
    "noUncheckedIndexedAccess": true
  }
}

At the time of writing, Deno does not automatically detect a deno.json file so it must be specified via the --config flag. However, this feature is planned for a future release.

$ deno run --config deno.json index.ts

When the Deno CLI encounters a type error, it halts the compilation of the script and terminates with a non-zero exit code. You can bypass the error by:

using //@ts-ignore or //@ts-expect-error at the point where the error occurred or
// @ts-nocheck at the beginning of the file to ignore all errors in a file.

Deno also provides a --no-check flag to disable type checking altogether. This helps prevent the TypeScript compiler from slowing you down when iterating quickly on a problem.

$ deno run --no-check index.ts

Permissions in Deno

Deno prides itself on being a secure runtime for JavaScript and TypeScript. Part of the way it maintains security is through its permissions system. To demonstrate how permissions work in Deno, add the below script to your index.ts file. It's a script that fetches the latest global Covid-19 statistics from disease.sh.

async function getCovidStats() {
  try {
    const response = await fetch("https://disease.sh/v3/covid-19/all");
    const data = await response.json();
    console.table(data);
  } catch (err) {
    console.error(err);
  }
}

getCovidStats();

When you attempt to execute the script, it should display a PermissionDenied error:

$ deno run index.ts
PermissionDenied: Requires net access to "disease.sh", run again with the --allow-net flag

The error message above indicates that the script hasn't been granted network access. It suggests including the --allow-net flag in the command to grant access.

$ deno run --allow-net index.ts
┌────────────────────────┬───────────────┐
│ (idx)                  │ Values        │
├────────────────────────┼───────────────┤
│ updated                │ 1633335683059 │
│ cases                  │     235736138 │
│ todayCases             │         32766 │
│ deaths                 │       4816283 │
│ todayDeaths            │           670 │
│ recovered              │     212616434 │
│ todayRecovered         │         51546 │
│ active                 │      18303421 │
│ critical               │         86856 │
│ casesPerOneMillion     │         30243 │
│ deathsPerOneMillion    │         617.9 │
│ tests                  │    3716763329 │
│ testsPerOneMillion     │     473234.63 │
│ population             │    7853954694 │
│ oneCasePerPeople       │             0 │
│ oneDeathPerPeople      │             0 │
│ oneTestPerPeople       │             0 │
│ activePerOneMillion    │       2330.47 │
│ recoveredPerOneMillion │      27071.26 │
│ criticalPerOneMillion  │         11.06 │
│ affectedCountries      │           223 │
└────────────────────────┴───────────────┘

Instead of granting blanket approval for the script to access all websites (as shown above), you can provide an allowlist of comma-separated hostnames or IP addresses as an argument to --allow-net so that only the specified websites are accessible by the script. If the script tries to connect to a domain that is not in the allowlist, Deno will prevent it from connecting, and the script execution will fail.

$ deno run --allow-net='disease.sh' index.ts

This feature is one of Deno's improvements over Node.js where
any script can access any resource over the network. Similar permissions also exist for reading from and writing to the filesystem. If a script needs to perform either task, you need to specify the --allow-read and --allow-write permissions, respectively. Both flags allow you to set the specific directories accessible to a script so that other parts of the filesystem are safe from tampering. Deno also provides an --allow-all flag that enables all security-sensitive functions for a script, if so desired.

Deno's Compatibility With Browser APIs

One of Deno's main goals is to be compatible with web browsers, where possible. This is reflected in its use of web platform APIs instead of creating a Deno-specific API for certain operations. For example, we saw the Fetch API in action in the previous section. This is the exact Fetch API used in browsers, with a few deviations where necessary to account for the unique security model in Deno (and these changes are mostly inconsequential).

There's a list of all implemented browser APIs in Deno's online documentation.

Dependency Management in Deno

The way Deno manages dependencies is probably the most obvious way it diverges significantly from Node.js.

Node.js uses a package manager like npm or yarn to download third-party packages from the npm registry into a node_modules directory and a package.json file to keep track of a project's dependencies. Deno does away with those mechanisms in favor of a more browser-centric way of using third-party packages: URLs.

Here's an example that uses Oak, a web application framework for Deno, to create a basic web server:

import { Application } from "https://deno.land/x/oak/mod.ts";

const app = new Application();

app.use((ctx) => {
  ctx.response.body = "Hello Deno!";
});

app.addEventListener("listen", ({ hostname, port, secure }) => {
  console.log(`Listening on: http://localhost:${port}`);
});

await app.listen({ port: 8000 });

Deno uses ES modules, the same module system used in web browsers. A module can be imported from an absolute or relative path, as long as the referenced script exports methods or other values. It's worth noting that the file extension must always be present, regardless of whether you import from an absolute or relative path.

While you can import modules from any URL, many third-party modules specifically built for Deno are cached through deno.land/x. Each time a new version of a module is released, it is automatically cached at that location and made immutable so that the contents of a specific version of a module remain unchangeable.

Suppose you run the code in the previous snippet. In that case, it will download the module and all its dependencies and cache them locally in the directory specified by the DENO_DIR environmental variable (the default should be $HOME/.cache/deno). The next time the program runs, there will be no downloads since all the dependencies have been cached locally. This is similar to how the Go module system works.

$ deno run --allow-net index.ts
Download https://deno.land/x/oak/mod.ts
Warning Implicitly using latest version (v9.0.1) for https://deno.land/x/oak/mod.ts
Download https://deno.land/x/oak@v9.0.1/mod.ts
. . .

For production applications, the creators of Deno recommend vendoring your dependencies by checking them into source control to ensure continued availability (even if the source of the module is unavailable, for whatever reason). Point the DENO_DIR environmental variable to a local directory in your project (such as vendor), which you can commit to Git.

For example, the command below will download all your script's dependencies into a vendor directory in your project. You can subsequently commit the folder to pull it down all at once in your production server. You'll also need to set the DENO_DIR variable to read from the vendor directory on the server, instead of downloading them all over again.

$ DENO_DIR=$PWD/vendor deno cache index.ts # Linux and macOS
$ $env:DENO_DIR="$(get-location)\vendor"; deno cache index.ts # Windows PowerShell

Deno also supports the concept of versioning your dependencies to ensure reproducible builds. At the moment, we've imported Oak from https://deno.land/x/oak/mod.ts. This always downloads the latest version, which could become incompatible with your program in the future. It also causes Deno to produce a warning when you download the module for the first time:

Warning Implicitly using latest version (v9.0.1) for https://deno.land/x/oak/mod.ts

It is considered best practice to reference a specific release as follows:

import { Application } from 'https://deno.land/x/oak@v9.0.1/mod.ts';

If you're referencing a module in many files in your codebase, upgrading it may become tedious since you have to update the URL in many places. To circumvent this issue, the Deno team recommends importing your external dependencies in a centralized deps.ts file and then re-exporting them. Here's a sample deps.ts file that exports what we need from the Oak library.

export { Application, Router } from "https://deno.land/x/oak@v9.0.1/mod.ts";

Then in your application code, you can import them as follows:

import { Application, Router } from "./deps.ts";

At this point, updating a module becomes a simple matter of changing the URL in the deps.ts file to point to the new version.

The Deno Standard Library

Deno provides a standard libary (stdlib) that aims to be a loose port of Go's standard library. The modules contained in the standard library are audited by the Deno team and updated with each release of Deno. The intention behind providing a stdlib is to allow you to create useful web applications right away, without resorting to any third-party packages (as is the norm in the Node.js ecosystem).

Some examples of standard library modules you might find helpful include:

HTTP: An HTTP client and server implementation for Deno.
Fmt: Includes helpers for printing formatted output.
Testing: Provides basic utilities for testing and benchmarking your code.
FS: Has helpers for manipulating the filesystem.
Encoding: Provides helpers to deal with various file formats, such as XML, CSV, base64, YAML, binary, and more.
Node: Has a compatibility layer for the Node.js standard library.

Here's an example (taken from Deno's official docs) that utilizes the http module in Deno's stdlib to create a basic web server:

import { listenAndServe } from "https://deno.land/std@0.109.0/http/server.ts";

const addr = ":8080";

const handler = (request: Request): Response => {
  let body = "Your user-agent is:\n\n";
  body += request.headers.get("user-agent") || "Unknown";

  return new Response(body, { status: 200 });
};

console.log(`HTTP webserver running. Access it at: http://localhost:8080/`);
await listenAndServe(addr, handler);

Start the server through the command below:

$ deno run --allow-net index.ts
HTTP webserver running. Access it at: http://localhost:8080/

In a different terminal, access the running server through the following command:

$ curl http://localhost:8080
Your user-agent is:

curl/7.68.0

Note that the modules in the standard library are currently tagged as unstable (as reflected in the version number). This means you should not rely on them just yet for a serious production application.

Using NPM Packages in Deno

It cannot be denied that one of the major reasons why Node.js has been so successful is the large number of packages that can be downloaded and utilized in a project. If you're considering switching to Deno, you may be wondering if you'd have to give up all the NPM packages you know and love.

The short answer is: no. While you may not be able to utilize some NPM packages in Deno if they rely on Node.js APIs (especially if the specific APIs are not supported in Deno's Node.js compatibility layer), many NPM packages can be utilized in Deno through CDNs like
esm.sh and skypack.dev. Both these CDNs provide NPM packages as ES Modules that can be subsequently consumed in a Deno script even if the author of the package did not design it to target Deno specifically.

Here's an example that imports the dayjs NPM package from Skypack in a Deno script:

import dayjs from "https://cdn.skypack.dev/dayjs@1.10.7";

console.log(`Today is: ${dayjs().format("MMMM DD, YYYY")}`);

$ deno run index.ts
Today is: October 05, 2021

To ensure that Deno can discover the types associated with a package, ensure you add the ?dts suffix at the end of the package URL when using Skypack's CDN. This causes Skypack to set a X-TypeScript-Types header so that Deno can automatically discover the types associated with a package. Esm.sh includes this header by default, but you can opt-out by adding the ?no-check suffix at the end of the package URL.

import dayjs from "https://cdn.skypack.dev/dayjs@1.10.7?dts";

Deno Tooling

The Deno CLI comes with several valuable tools that make the developer experience much more pleasant. Like Node.js, it comes with a REPL (Read Evaluate Print Loop), which you can access with deno repl.

$ deno repl
Deno 1.14.2
exit using ctrl+d or close()
> 2+2
4

It also has a built-in file watcher that can be used with several of its subcommands. For example, you can configure deno run to automatically rebuild and restart a program once a file is changed by using the --watch flag. In Node.js, this functionality is generally achieved through some third-party package such as nodemon.

$ deno run --allow-net --watch index.ts
HTTP webserver running. Access it at: http://localhost:8080/
Watcher File change detected! Restarting!
HTTP webserver running. Access it at: http://localhost:8080/

With Deno 1.6, you can compile scripts into self-contained executables that do not require Deno to be installed through the compile subcommand (you can use pkg to do the same in Node.js). You can also generate executables for other platforms (cross compilation) through the --target flag. When compiling a script, you must specify the permissions needed for it to run.

$ deno compile --allow-net --output server index.ts
$ ./server
HTTP webserver running. Access it at: http://localhost:8080/

Note that the binaries produced through this process are quite huge. In my testing, deno compile produced an 83MB binary for a simple "Hello world" program. However, the Deno team is currently working on a way to reduce the file sizes to be a lot more manageable.

Another way to distribute a Deno program is to package it into a single JavaScript file through the bundle subcommand. This file contains the source code of the program and all its dependencies, and it can be executed through deno run as shown below.

$ deno bundle index.ts index.bundle.js
Check file:///home/ayo/dev/demo/deno/index.js
Bundle file:///home/ayo/dev/demo/deno/index.js
Emit "index.bundle.js" (7.39KB)

$ deno run --allow-net index.bundle.js
HTTP webserver running. Access it at: http://localhost:8080/

Two additional great tools that Deno ships with are the built-in linter (deno lint) and formatter (deno fmt). In the Node.js ecosystem, linting and formatting code are typically handled with ESLint and Prettier, respectively.

When using Deno, you no longer need to install anything or write configuration files to get linting and formatting for JavaScript, TypeScript, and other supported file formats.

Unit Testing in Deno

Support for unit testing is built into Deno for both JavaScript and TypeScript code. When you run deno test, it automatically detects any files that end with _test.ts or .test.ts (also supports other file extensions) and executes any defined tests therein.

To write your first test, create an index_test.ts file and populate it with the following code:

import { assertEquals } from "https://deno.land/std@0.109.0/testing/asserts.ts";

Deno.test("Multiply two numbers", () => {
  const ans = 2 * 2;
  assertEquals(ans, 4);
});

Deno provides the Deno.test method for creating a unit test. It takes the name of the test as its first argument. Its second argument is the function executed when the test runs.

There is a second style that takes in an object instead of two arguments. It supports other properties aside from the test name and function to configure if or how the test should run.

Deno.test({
  name: "Multiply two numbers",
  fn() {
    const ans = 2 * 2;
    assertEquals(ans, 4);
  },
});

The assertEquals() method comes from the testing module in the standard library, and it provides a way to easily check the equality of two values.

Go ahead and run the test:

$ deno test
test Multiply two numbers ... ok (8ms)

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out (37ms)

The Deno Language Server

One of the major considerations for choosing a programming language or environment is its integration with editors and IDEs. In Deno 1.6, a built-in language server (deno lsp) was added to the runtime to provide features such as:

autocompletion
go-to-definition
linting and formatting integration

As well as other language smarts for any editor that supports the Language Server Protocol (LSP). You can learn more about setting up Deno support in your editor in Deno's online docs.

Wrapping up: Should I Choose Deno over Node.js?

In this article, we've considered many aspects of the Deno runtime, and ways in which it's an upgrade over Node.js.

There’s a lot more to say about Deno and its ecosystem, but this should hopefully be a helpful introduction for Node.js developers considering Deno for a new project. The lesser availability of third-party packages for Deno is an obvious aspect where it falls short, as is the fact that it's not as battle-tested as Node.js in the real world due to its young age (Deno 1.0 was released in May 2020).

Comparing the performance between Node.js and Deno shows that they're within the same ballpark in most cases, although there are a few scenarios where Node.js exhibits far superior performance. The measured disparities are bound to improve as Deno becomes more mature.

When deciding between Node.js and Deno, it's also important to keep in mind that some of the benefits that Deno provides can also be brought to Node.js with the help of third-party packages. So if there are only one or two things that you admire about Deno, chances are you'll be able to achieve a similar result in Node.js, though not as seamlessly.

Thanks for reading, and happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

How to Set Up a Node.js Project with TypeScript

Ayooluwa Isaiah — Wed, 26 Jan 2022 13:18:00 +0000

In this tutorial, you will learn how to add TypeScript support to Node.js projects. We will address common needs, such as:

compiling and executing the code
debugging the source files
configuring third-party packages so that the TypeScript compiler also validates them

Let's get going!

Why Use TypeScript?

TypeScript brings optional static typing to JavaScript projects. The primary benefit of static typing is that type errors are detected and corrected at build time, so code will more likely run correctly once deployed to production. A growing number of JavaScript developers are seeing the value of writing more strongly typed code, which has led to an increase in the adoption of TypeScript for all kinds of JavaScript-based projects.

Prerequisites

This article assumes that you have a basic knowledge of Node.js and TypeScript. You also need to have a recent version of Node.js and npm installed on your computer.

Installing and Configuring TypeScript on Node.js

Besides the web browser, Node.js is the most popular platform on which JavaScript code is executed. However, just like the browser, it lacks native support for TypeScript code (unlike Deno, for example).

Before installing TypeScript, make sure that you've created and initialized a Node.js project with a package.json file. TypeScript is available as a package on the npm registry, and it can be downloaded into your project through a package manager like npm or yarn:

$ npm install typescript --save-dev

Once the above command succeeds, you can check the current version through the following command:

$ npx tsc --version
Version 4.4.3

It's advisable to install TypeScript as a project-specific dependency. The same version of the language will then be used, regardless of the machine running the code. You can also install the TypeScript CLI globally by using the --global switch. A global installation can come in handy for running one-off scripts for testing purposes.

$ npm install typescript --global
$ tsc --version
Version 4.4.3

If you use a Node.js environment manager like Volta, you'll be able to switch between globally installed TypeScript versions and project-specific ones seamlessly.

Now that TypeScript is installed in your project, create a configuration file that specifies which files should be compiled and the compiler options for the project. This file is called tsconfig.json, and you should place it at the root of your project directory.

$ touch tsconfig.json

Here's a basic configuration that you can start with:

{
  "extends": "@tsconfig/node16/tsconfig.json",
  "compilerOptions": {},
  "include": ["src"],
  "exclude": ["node_modules"]
}

The above configuration file extends the base configuration provided by the TypeScript team for Node.js v16. Additional options or overrides may be included through the compilerOptions property. It also specifies that all the files in the src directory should be included in the program, but everything in the node_modules directory is skipped entirely. Both the include and exclude properties support glob patterns.

Before you proceed, ensure you add the base configuration package for Node.js v16 to your project's devDependencies through the command below. Base tsconfig.json packages also exist for Node 10, Node 12, and Node 14 at the time of writing.

$ npm install @tsconfig/node16 --save-dev

Compiling TypeScript Files for Node.js

Go ahead and create the aforementioned src directory in your project root, and place a main.ts file inside it. This file should contain the following code:

function sayMyName(name: string): void {
  if (name === "Heisenberg") {
    console.log("You're right 👍");
  } else {
    console.log("You're wrong 👎");
  }
}

sayMyName("Heisenberg");

Save the file, then attempt to compile the TypeScript code to JavaScript
through the command below:

$ npx tsc

You will get an error indicating that the compiler does not understand the console object:

src/main.ts:3:5 - error TS2584: Cannot find name 'console'. Do you need to change your target library? Try changing the 'lib' compiler option to include 'dom'.
src/main.ts:5:1 - error TS2584: Cannot find name 'console'. Do you need to change your target library? Try changing the 'lib' compiler option to include 'dom'.

5 console.log(product(10, 5));
  ~~~~~~~

Found 1 error.

3     console.log("You're right 👍");
      `~~~~~~~`

src/main.ts:5:5 - error TS2584: Cannot find name 'console'. Do you need to change your target library? Try changing the 'lib' compiler option to include 'dom'.

5     console.log("You're wrong 👎");
      `~~~~~~~`


Found 2 errors.

This error occurs because the lib compiler option set in the base configuration for Node.js v16 does not include the dom option, which contains type definitions for the console object and other browser-specific APIs. The error message above suggests adding the dom option to the lib property to fix the problem, but this is not the correct solution for a Node.js project. The correct fix involves installing the type definitions for Node.js APIs so that the TypeScript compiler can understand and validate all the built-in Node.js APIs.

Here's how:

$ npm install @types/node --save-dev

Once installed, the error will vanish the next time npx tsc is run and a main.js file will be produced in the src folder with the following content:

"use strict";
function sayMyName(name) {
    if (name === "Heisenberg") {
        console.log("You're right 👍");
    }
    else {
        console.log("You're wrong 👎");
    }
}
sayMyName("Heisenberg");

You can subsequently execute this file through the node command:

$ node src/main.js
You're right 👍

If you want to change the folder where the JavaScript files are placed, you can use the outDir compiler option in your tsconfig.json file:

{
  "compilerOptions": {
    "outDir": "dist"
  }
}

Subsequent compilations will emit the JavaScript output into a dist folder.

.
├── dist
│   └── main.js
├── package.json
├── package-lock.json
├── src
│   └── main.ts
└── tsconfig.json

Execute TypeScript Source Files Directly With ts-node

The process of compiling TypeScript source files into JavaScript code before executing them with Node.js can get a little tedious after a while, especially during development. You can eliminate the intermediate steps before running the program through the ts-node CLI to execute .ts files directly. Go ahead and install the ts-node package using the command below:

$ npm install ts-node --save-dev

Afterwards, execute the main.ts file with the ts-node command:

$ npx ts-node src/main.ts
You're right 👍

Using ts-node in this way places the TypeScript compiler as a middleman between the source files and the Node.js runtime. It transpiles the source code before executing the resulting JavaScript code with node (performed under the hood). This makes the script execution a bit slower than JavaScript output directly with node. You can opt out of type checking through the --transpile-only or -T flag to make the script execute faster in scenarios where type validation isn't essential.

Another feature that ts-node enables is the transformation of modern import syntax to CommonJS syntax. This means that when using ts-node, you can use import instead of require to utilize Node.js modules in your code.

Learn more about this feature in the ts-node project's README document.

TypeScript Integration with Third-party Node.js NPM Packages

When utilizing Node.js packages from the npm registry, additional setup may be required to compile the project successfully.

The main problem is that most of the packages you're likely to encounter are written in vanilla JavaScript, so TypeScript cannot determine the valid types for exposed methods. Everything from the library is implicitly typed as any.

Here's an example that utilizes the popular Express package to create a web server:

import express from 'express';
const app = express();

app.get('/', function (req, res) {
  res.send('Hello World');
})

app.listen(3000);

Assuming you've installed the express module with npm install express, execute the script with ts-node. It should yield the following errors:

$ npx ts-node src/main.ts
TSError: ⨯ Unable to compile TypeScript:
src/main.ts:4:24 - error TS7006: Parameter 'req' implicitly has an 'any' type.

4 app.get('/', function (req, res) {
                         ~~~

src/main.ts:4:29 - error TS7006: Parameter 'res' implicitly has an 'any' type.

4 app.get('/', function (req, res) {
                              ~~~


Found 2 errors.

The TypeScript compiler is responsible for the errors shown above. It cannot determine the type of the req and res parameters in the callback function, so they are both implicitly typed as any.

Since the strict compiler option is set to true in the base tsconfig.json file, the noImplicitAny compiler option is also enabled. This ensures that TypeScript will emit an error instead of inferring type any when it is unable to determine the type of a value.

You can fix this error by providing a type declaration file for the express module so that the TypeScript compiler can accurately determine the valid types for its exported methods. You can find up-to-date type definitions for many popular npm packages in the DefinitelyTyped GitHub repository.

The type definition for a package can be downloaded through the @types scope. For example, use the command below to install the type definitions for Express:

$ npm install @types/express --save-dev

After installing the type definitions for Express, compiling and executing the script with ts-node should complete successfully without errors.

Linting TypeScript with ESLint

An important step towards adding comprehensive support for TypeScript in a Node.js application is setting an adequate linting workflow. You can make use of the popular ESLint package to lint TypeScript code. Although it was originally written for JavaScript code, it also supports TypeScript with the help of a few plugins. Go ahead and install the eslint package in your project with the command below:

$ npm install eslint --save-dev

Now create a new .eslintrc.js file in the root of your project directory. Here is where you'll place the configuration settings for ESLint. To add TypeScript support to ESLint, install the @typescript-eslint/parser and @typescript-eslint/eslint-plugin packages in your project. The former is used to parse TypeScript code to a format that is understandable by ESLint, while the latter provides TypeScript-specific linting rules.

$ npm install @typescript-eslint/parser @typescript-eslint/eslint-plugin --save-dev

Once both packages are installed, open up your .eslintrc.js file in your editor, and enter the following:

module.exports = {
  parser: '@typescript-eslint/parser',
  parserOptions: {
    ecmaVersion: 'latest', // Allows the use of modern ECMAScript features
    sourceType: 'module', // Allows for the use of imports
  },
  extends: ['plugin:@typescript-eslint/recommended'], // Uses the linting rules from @typescript-eslint/eslint-plugin
  env: {
    node: true, // Enable Node.js global variables
  },
};

If you want to override any of the linting rules or configure other rules, use the rules property in the .eslintrc.js file, as shown below:

module.exports = {
  . . .
  rules: {
    'no-console': 'off',
    'import/prefer-default-export': 'off',
    '@typescript-eslint/no-unused-vars': 'warn',
  },
};

After saving the file, run the ESLint CLI on your TypeScript code through this command:

$ npx eslint . --fix

You can also create a lint script in your package.json file as follows:

{
  "scripts": {
    . . .
    "lint": "eslint . --fix"
  }
}

Then run ESLint:

$ npm run lint

To prevent ESLint from linting certain files or directories, create a .eslintignore file in your project root, and place the patterns for files to ignore therein. Here's an example in which all generated files in the dist folder are ignored:

dist

You can also decide to omit every single JavaScript file in your project directory with the following pattern:

**/*.js

Note that everything in the node_modules folder, and files or folders that begin with a dot character (except eslint config files), are ignored automatically, so there's no need to place patterns matching such files in your .eslintignore file.

Debug Your TypeScript Code with Visual Studio Code

Debugging TypeScript source files is easy and straightforward with the help of ts-node and Visual Studio Code. All you need to do is create a launch configuration file (launch.json) within the .vscode directory in the project root and populate the file with the following code:

{
  "version": "0.1.0",
  "configurations": [
    {
      "name": "Debug main.ts",
      "type": "node",
      "request": "launch",
      "cwd": "${workspaceRoot}",
      "runtimeArgs": ["-r", "ts-node/register"],
      "args": ["${workspaceRoot}/src/main.ts"]
    }
  ]
}

The ts-node/register method is preloaded in the above file to handle TypeScript source files correctly. Secondly, the name of the TypeScript file to run when starting a debugging session is provided as the first value in the args property.

Go ahead and start debugging your Node.js project by pressing F5 on your keyboard. Try to set a breakpoint, then inspect the values in the current scope once the breakpoint is hit. It should work as expected, without issues!

Deploying TypeScript to Production

According to its author, ts-node is safe to use in production. Nonetheless, to reduce the start-up time of the server and prevent additional memory usage from keeping the compiler in memory, it's better to compile the source files beforehand. Execute the resulting JavaScript code with the node command when deploying to production.

Node.js and TypeScript: Summing Up

In this article, you learned how to configure a Node.js project with TypeScript support and run TypeScript source files directly, without an intermediate compilation step.

We also covered how type definition files work, and how to take advantage of predefined type definition files for popular npm packages so that the TypeScript compiler fully validates all third-party dependencies.

Finally, we discussed how to debug TypeScript files in VSCode, and what to do when deploying your TypeScript project to production.

Thanks for reading, and happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

7 Ways to Improve Node.js Performance at Scale

Ayooluwa Isaiah — Wed, 08 Dec 2021 13:00:03 +0000

Performance is one of the most important aspects of web application development.

A fast application will make its users, developers, and business stakeholders happy, while a slow one is sure to frustrate all three parties.

In this article, we will consider some practices that you should adopt to scale your Node.js servers. Your servers will then be able to handle high traffic workloads without a degraded user experience.

By following all of the proven and tested performance tips in this post, you will be able improve the speed and performance of your product, giving it the edge it needs to succeed in the market.

1. Profile and Monitor Your Application

Before you attempt to improve the performance of a system, it's necessary to measure the current level of performance. This way, you'll know the inefficiencies and the right strategy to adopt to get the desired results.

Gauging the current level of performance of an application may require running different kinds of tests, such as the following:

Load testing: refers to the practice of simulating the expected usage of a system and measuring its response as the workload increases.
Stress testing: designed to measure how a system performs beyond the limits of normal working conditions. Its goal is to determine how much the system can handle before it fails and how it attempts to recover from a failure.
Spike testing: helps to test the behavior of an application when it receives a drastic increase or decrease in load.
Scalability testing: used to find the point at which the application stops scaling and identify the reasons behind it.
Volume testing: determines if a system can cope with large amounts of data.
Endurance testing: helps evaluate the behavior of a software application under sustained load for a long period, to catch problems such as memory leaks.

Performing some or all of the above tests will provide you with several important metrics, such as:

response times
average latency
error rates
requests per second
throughput
CPU and memory usage
concurrent users

and more.

After implementing a specific optimization, don't forget to rerun the tests to verify that your changes had the desired effect on system performance.

It's also important to utilize an Application Performance Monitoring (APM) tool to keep tabs on the long-term performance of a system. Different monitoring solutions can take care of this this for you. We like AppSignal :).

It's easy to integrate it into your application (just run npx @appsignal/cli install), and it will automatically track several performance metrics such as response times and throughput alongside error logs, system availability, host metrics, and more. You can use the insights gained from the data to take proactive steps to increase system performance or to quickly identify the root cause of a specific issue so that you can address it promptly before it is noticed by your users.

2. Reduce Latency Through Caching

Server-side caching is one of the most common strategies for improving the performance of a web application. Its primary aim is to increase the speed of data retrieval, either by spending less time computing such data or doing I/O (such as retrieving such data over the network or from a database).

A cache is a high-speed storage layer used as a temporary store for frequently accessed data. You don't have to retrieve data from the (usually much slower) primary source of the data every time it is requested.

Caching is most effective for data that does not change very often. If your application is receiving lots of requests for the same unchanged data, storing it in a cache is sure to significantly improve the responsiveness of such requests. You can also store the results of computationally intensive tasks in the cache, as long as it can be reused for other requests. This prevents server resources from being bogged down unnecessarily by repeating the work to compute such data.

Another common candidate for caching is API requests that go to an external system. Suppose the responses can be reliably reused for subsequent requests. In that case, it makes sense to store API requests in the cache layer to avoid the additional network request and any other costs associated with the API in question.

A relatively straightforward way to implement caching in a Node.js application is through an in-process caching solution such as node-cache. It involves placing actively used data in memory, where it can be retrieved more quickly. The major problem with an in-process cache is that it's tied to an application process, so it's rarely suitable for distributed workflows (especially when caching mutable objects). In such settings, you may use a distributed caching solution such as Redis or Memcached. These run independently of an application and are more practical when scaling the application onto multiple servers.

3. Use Timeouts When Dealing with I/O Operations

When building Node.js applications, timeouts are amongst the easiest things to get wrong. Your server is probably talking to other external services that themselves might also be calling other services. If one service in the chain is slow or unresponsive, it will result in a slow experience for your end-users. Even if you don't run into this issue during development, you can't guarantee that your dependencies will always respond as fast as they usually do, which is why the concept of timeouts is important.

A timeout is the maximum wait time set on a request. It represents how long a client is prepared to wait for a response from an external service. If the response is not received within the specified limits, the connection will abort so that the application does not hang indefinitely. Many popular libraries for making HTTP requests in Node.js (such as axios) do not set a default timeout, meaning that any remote API can keep your application waiting for the requested resource indefinitely. You should set a request timeout to prevent this from happening:

const axios = require('axios');

axios.defaults.timeout === 1000; // global timeout of 1s

In the above snippet, a timeout of 1000ms (1s) is set as the default for all HTTP requests made through axios. This guarantees that any request will not take longer than that time, even if the API is unresponsive. You can also set a timeout value on individual requests when the global default is not appropriate:

axios
  .get('https://example.com/api', { timeout: 2000 })
  .then(response => {
     console.log(response);
  })
  .catch(error => {
     console.log(error);
  });

Note that the axios timeout value is a read timeout, which is different from a connection timeout. The latter is the time within which a TCP connection must be established, while the former determines how long the client will wait for a response after the connection is established.

Usually, the connection timeout is much lower than the read timeout. The client is able to try a different server or alternative API if one service is taking too long to accept a connection. This still gives enough time for the server to generate a response once the connection is accepted.

At the moment, axios does not support setting a connection timeout separately from a read timeout, which could be limiting in some scenarios. If you need this functionality, you can try the got library - it allows for separate read and connection timeout
specifications.

Before choosing a timeout value, you can monitor the response times for APIs you connect to using specialized tools or track your API calls by logging them. This will allow you to make an informed decision for all the external services that your program interacts with. You should also have a retry strategy in place for important services to account for temporary slowdowns. The graph below shows how average the response times for an endpoint can be monitored in AppSignal.

4. Don't Serve Static Assets with Node.js

To ensure the best possible performance for your Node.js servers, refrain from using them to serve static assets like JavaScript, CSS, or image files from your application. Node.js wasn't designed with this use case in mind, so serving assets from the main application consumes valuable resources and holds up important business computations. Offload the task of serving static files to a web server like Nginx, which can perform optimizations that does not make sense for Node.js to do. This test shows that Nginx is about twice as fast at delivering static assets as Node.js (using Express static middleware).

Another option to serve static files is to set up a CDN proxy like Amazon CloudFront to cache your static content and serve it as close as possible to end-users. This frees up the Node.js servers to handle dynamic requests only.

5. Use Clustering to Improve Throughput

Clustering is a technique used to horizontally scale a Node.js server on a single machine by spawning child processes (workers) that run concurrently and share a single port. It is a common tactic to reduce downtime, slowdowns and outages by distributing the incoming connections across all the available worker processes so that available CPU cores are utilized to their full potential. Since a Node.js instance runs on a single thread, it cannot take advantage of multi-core systems properly - hence the need for clustering.

You can cluster your Node.js server through the cluster module in the standard library. Here's an example taken from the official documentation:

const cluster = require('cluster');
const http = require('http');
const process = require('process');
const os = require('os');

const cpus = os.cpus;

const numCPUs = cpus().length;

if (cluster.isPrimary) {
  console.log(`Primary ${process.pid} is running`);

  // Fork workers.
  for (let i = 0; i < numCPUs; i++) {
    cluster.fork();
  }

  cluster.on('exit', (worker, code, signal) => {
    console.log(`worker ${worker.process.pid} died`);
  });
} else {
  // Workers can share any TCP connection
  // In this case it is an HTTP server
  http
    .createServer((req, res) => {
      res.writeHead(200);
      res.end('hello world\n');
    })
    .listen(8000);

  console.log(`Worker ${process.pid} started`);
}

Once you run this program, connections sent to port 8000 will be shared between the worker processes. This will lead to more efficient requests management in the application:

$ node server.js
Primary 15990 is running
Worker 15997 started
Worker 15998 started
Worker 16010 started
Worker 16004 started

A downside to using the native cluster module in Node.js is the amount of code you need to write to spawn and manage the workers, and it is not possible to modify the number of processes on the fly.

For more robust management of Node.js clusters, use the PM2 process manager for Node.js. It uses the cluster module under the hood and takes care of spawning workers, stopping or restarting workers, and distributing the incoming load between the workers. It also provides some tools to help you monitor and tweak the performance of worker processes:

6. Scale across Multiple Machines with a Load Balancer

Scaling your Node.js application horizontally across multiple machines is similar to scaling across multiple cores on a single machine. As long as your application can run as an independent process, it can be distributed to run across several machines. The major requirement is using a load balancer to distribute incoming traffic to the servers (similar to how the cluster module is used to direct traffic to the child worker process). You can even have multiple load balancers that point to the same set of servers to avoid a single point of failure.

7. Utilize Worker Threads for CPU-intensive Tasks

Worker threads provide a mechanism to run CPU-intensive tasks in a Node.js application without blocking the main event loop. They were introduced in Node.js v10.5.0, and only became stable in the v12.0.0 release.

A worker thread is spawned by the main or parent thread, and its responsibility is to perform a task in isolation from other workers. Unlike child processes or clusters, worker threads can share memory by transferring ArrayBuffer instances or sharing SharedArrayBuffer instances. A worker and their parent can also communicate in both directions using a message channel.

Here's how to create a worker thread using the worker_threads module from the standard library:

// main.js
const { Worker } = require('worker_threads');

// Create a new worker
const worker = new Worker('./worker.js');

// Listen for messages from worker
worker.on('message', (result) => {
  console.log(
    `The prime numbers between 2 and ${result.input} are: ${result.primes}`
  );
});

worker.on('error', (error) => {
  console.log(error);
});

worker.on('exit', (exitCode) => {
  console.log(exitCode);
});

// Send messages to the worker
worker.postMessage({ input: 100 });
worker.postMessage({ input: 50 });

When main.js is executed, it spawns a new worker thread derived from the worker.js file. The postMessage() method sends messages to the worker, and a listener is used to detect responses from the worker. The worker.js file is shown below:

const { parent } = require('worker_threads');

parent.on('message', (data) => {
  parent.postMessage({
    input: data.input,
    primes: getPrimes(data.input),
  });
});

function getPrimes(max) {
  const sieve = [],
    primes = [];

  for (let i = 2; i <= max; ++i) {
    if (!sieve[i]) {
      primes.push(i);

      for (let j = i << 1; j <= max; j += i) {
        sieve[j] = true;
      }
    }
  }

  return primes;
}

In the above snippet, the getPrimes() function is used to find all the prime numbers between 2 and the specified argument which is received from the parent through the message listener. The result is also sent back to the parent by using the postMessage() method as before:

The prime numbers between 2 and 100 are: 2,3,5,7,11,13,17,19,23,29,31,37,41,43,47,53,59,61,67,71,73,79,83,89,97
The prime numbers between 2 and 50 are: 2,3,5,7,11,13,17,19,23,29,31,37,41,43,47

Read the official documentation of the worker_threads
module to learn more about using Node.js workers to your advantage.

Additional Tips to Improve Node.js Performance

Here are some micro-optimizations you can make in your Node.js application to reliably get better results:

Always use the latest release of Node.js to get the best possible performance.
Pay attention to your dependencies, and choose the most performant libraries where possible. Sometimes, it is better to forgo adding a dependency and instead write the code to perform a task yourself.
Ensure that all independent I/O operations use asynchronous primitives like callbacks, promises, and async/await to ensure a non-blocking operation flow and improve downstream latency.
You don't have to optimize everything. Once the hotspots of your application are well optimized, stop.
Your hot spots may change over time, so make sure to use some form of observability or monitoring solution to track these changes.
When working with large pieces of data, use Node.js streams for optimal
memory efficiency and reduced latency.
To reduce the load on the garbage collector (thus reducing latency), avoid memory allocations in hotspots.
Optimize your database queries, and scale them appropriately to ensure that they do not become a bottleneck.
Don't trade performance for reliability. Try to strike a balance between tuning your code for performance, the cost of development, and continued maintenance.

Wrap-up: Scale Your Node.js Application by Improving Performance

In this article, we've covered some practical tips to help you scale your Node.js application to handle more traffic. Before implementing a specific optimization, ensure you run comprehensive performance tests on your system and use the insights you gain to determine what course of action to take. Also, use observability/monitoring tools so that you can see the impact of your changes and spot regressions quickly and reliably.

If you have any additional tips regarding performance optimization in Node.js that were not discussed in this post, feel free to share them with me on Twitter.

Thanks for reading, and happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Our guest author Ayo is a Software Developer by trade. He enjoys writing about diverse technologies in web development, mainly in Go and JavaScript/TypeScript. You can learn more about him through his blog.

A Guide to Load Testing Node.js APIs with Artillery

Ayooluwa Isaiah — Wed, 24 Nov 2021 12:43:18 +0000

Artillery is an open-source command-line tool purpose-built for load testing and smoke testing web applications. It is written in JavaScript and it supports testing HTTP, Socket.io, and WebSockets APIs.

This article will get you started with load testing your Node.js APIs using Artillery. You'll be able to detect and fix critical performance issues before you deploy code to production.

Before we dive in and set up Artillery for a Node.js app, though, let's first answer the question: what is load testing and why is it important?

Why Should You Do Load Tests in Node.js?

Load testing is essential to quantify system performance and identify breaking points at which an application starts to fail. A load test generally involves simulating user queries to a remote server.

Load tests reproduce real-world workloads to measure how a system responds to a specified load volume over time. You can determine if a system behaves correctly under loads it is designed to handle and how adaptable it is to spikes in traffic. It is closely related to stress testing, which assesses how a system behaves under extreme loads and if it can recover once traffic returns to normal levels.

Load testing can help validate if an application can withstand realistic load scenarios without a degradation in performance. It can also help uncover issues like:

Increased response times
Memory leaks
Poor performance of various system components under load

As well as other design issues that contribute to a suboptimal user experience.

In this article, we'll focus on the free and open-source version of Artillery to explore load testing. However, bear in mind that a pro version of Artillery is also available for those whose needs exceed what can be achieved through the free version. It provides added features for testing at scale and is designed to be usable even if you don't have prior DevOps experience.

Installing Artillery for Node.js

Artillery is an npm package so you can install it through npm or yarn:

$ yarn global add artillery

If this is successful, the artillery program should be accessible from the command line:

$ artillery -V
        ___         __  _ ____                  _
  _____/   |  _____/ /_(_) / /__  _______  __  (_)___  _____
 /____/ /| | / ___/ __/ / / / _ \/ ___/ / / / / / __ \/____/
/____/ ___ |/ /  / /_/ / / /  __/ /  / /_/ / / / /_/ /____/
    /_/  |_/_/   \__/_/_/_/\___/_/   \__, (_)_/\____/
                                    /____/

------------ Version Info ------------
Artillery: 1.7.7
Artillery Pro: not installed (https://artillery.io/pro)
Node.js: v16.7.0
OS: linux/x64
--------------------------------------

Basic Artillery Usage

Once you've installed the Artillery CLI, you can start using it to send traffic to a web server. It provides a quick subcommand that lets you run a test without writing a test script first.

You'll need to specify:

an endpoint
the rate of virtual users per second or a fixed amount of virtual users
how many requests should be made per user

$ artillery quick --count 20 --num 10 http://localhost:4000/example

The --count parameter above specifies the total number of virtual users, while --num indicates the number of requests that should be made per user. Therefore, 200 (20*10) GET requests are sent to the specified endpoint. On successful completion of the test, a report is printed out to the console.

All virtual users finished
Summary report @ 14:46:26(+0100) 2021-08-29
  Scenarios launched:  20
  Scenarios completed: 20
  Requests completed:  200
  Mean response/sec: 136.99
  Response time (msec):
    min: 0
    max: 2
    median: 1
    p95: 1
    p99: 2
  Scenario counts:
    0: 20 (100%)
  Codes:
    200: 200

This shows several details about the test run, such as the requests completed, response times, time taken for the test, and more. It also displays the response codes received on each request so that you can determine if your API handles failures gracefully in cases of overload.

While the quick subcommand is handy for performing one-off tests from the command line, it's quite limited in what it can achieve. That's why Artillery provides a way to configure different load testing scenarios through test definition files in YAML or JSON formats. This allows great flexibility to simulate the expected flows at one or more of your application's endpoints.

Writing Your First Artillery Test Script

In this section, I'll demonstrate a basic test configuration that you can apply to any application. If you want to follow along, you can set up a test environment for your project, or run the tests locally so that your production environment is not affected. Ensure you install Artillery as a development dependency so that the version you use is consistent across all deployments.

$ yarn add -D artillery

An Artillery test script consists of two main sections: config and scenarios. config includes the general configuration settings for the test such as the target, response timeouts, default HTTP headers, etc. scenarios consist of the various requests that virtual users should make during a test. Here's a script that tests an endpoint by sending 10 virtual users every second for 30 seconds:

config:
  target: "http://localhost:4000"
  phases:
    - duration: 30
      arrivalRate: 10

scenarios:
  - name: "Retrieve data"
    flow:
      - get:
          url: "/example"

In the above script, the config section defines the base URL for the application that's being tested in the target property. All the endpoints defined later in the script will run against this base URL.

The phases property is then used to set up the number of virtual users generated in a period of time and how frequently these users are sent to specified endpoints.

In this test, duration determines that virtual users will be generated for 30 seconds and arrivalRate determines the number of virtual users sent to the endpoints per second (10 users).

On the other hand, the scenarios section defines the various operations that a virtual user should perform. This is controlled through the flow property, which specifies the exact steps that should be executed in order. In this case, we have a single step: a GET request to the /example endpoint on the base URL. Every virtual user that Artillery generates will make this request.

Now that we've written our first script, let's dive into how to run a load test.

Running a Load Test in Artillery

Save your test script to a file (such as load-test.yml) and execute it through the command below:

$ artillery run path/to/script.yml

This command will start sending virtual users to the specified endpoints at a rate of 10 requests per second. A report will be printed to the console every 10 seconds, informing you of the number of test scenarios launched and completed within the time period, and other statistics such as mean response time, HTTP response codes, and errors (if any).

Once the test concludes, a summary report (identical to the one we examined earlier) is printed out before the command exits.

All virtual users finished
Summary report @ 15:38:48(+0100) 2021-09-02
  Scenarios launched:  300
  Scenarios completed: 300
  Requests completed:  300
  Mean response/sec: 9.87
  Response time (msec):
    min: 0
    max: 1459
    median: 1
    p95: 549.5
    p99: 1370
  Scenario counts:
    Retrieve data: 300 (100%)
  Codes:
    200: 300

How to Create Realistic User Flows

The test script we executed in the previous section is not very different from the quick example in that it makes requests to only a single endpoint. However, you can use Artillery to test more complex user flows in an application.

In a SaaS product, for example, a user flow could be: someone lands on your homepage, checks out the pricing page, and then signs up for a free trial. You'll definitely want to find out how this flow will perform under stress if hundreds or thousands of users are trying to perform these actions at the same time.

Here's how you can define such a user flow in an Artillery test script:

config:
  target: "http://localhost:4000"
  phases:
    - duration: 60
      arrivalRate: 20
      name: "Warming up"
    - duration: 240
      arrivalRate: 20
      rampTo: 100
      name: "Ramping up"
    - duration: 500
      arrivalRate: 100
      name: "Sustained load"
  processor: "./processor.js"

scenarios:
  - name: "Sign up flow"
    flow:
      - get:
          url: "/"
      - think: 1
      - get:
          url: "/pricing"
      - think: 2
      - get:
          url: "/signup"
      - think: 3
      - post:
          url: "/signup"
          beforeRequest: generateSignupData
          json:
            email: "{{ email }}"
            password: "{{ password }}"

In the above script, we define three test phases in config.phases:

The first phase sends 20 virtual users per second to the application for 60 seconds.
In the second phase, the load will start at 20 users per second and gradually increase to 100 users per second over 240 seconds.
The third and final phase simulates a sustained load of 100 users per second for 500 seconds.

By providing several phases, you can accurately simulate real-world traffic patterns and test how adaptable your system is to a sudden barrage of requests.

The steps that each virtual user takes in the application are under scenarios.flow. The first request is GET / which leads to the homepage. Afterward, there is a pause for 1 second (configured with think) to simulate user scrolling or reading before making the next GET request to /pricing. After a further delay of 2 seconds, the virtual user makes a GET request to /signup. The last request is POST /signup, which sends a JSON payload in the request body.

The {{ email }} and {{ password }} placeholders are populated through the generateSignupData function, which executes before the request is made. This function is defined in the processor.js file referenced in config.processor. In this way, Artillery lets you specify custom hooks to execute at specific points during a test run. Here are the contents of processor.js:

const Faker = require('faker');

function generateSignupData(requestParams, ctx, ee, next) {
  ctx.vars['email'] = Faker.internet.exampleEmail();
  ctx.vars['password'] = Faker.internet.password(10);

  return next();
}

module.exports = {
  generateSignupData,
};

The generateSignupData function uses methods provided by Faker.js to generate a random email address and password each time it is called. The results are then set on the virtual user's context, and next() is called so that the scenario can continue to execute. You can use this approach to inject dynamic random content into your tests so they're as close as possible to real-world requests.

Note that other hooks are available aside from beforeRequest, including the following:

afterResponse - Executes one or more functions after a response has been received from the endpoint:

- post:
    url: "/login"
    afterResponse:
      - "logHeaders"
      - "logBody"

beforeScenario and afterScenario - Used to execute one or more functions before or after each request in a scenario:

scenarios:
  - beforeScenario: "setData"
    afterScenario: "logResults"
    flow:
      - get:
          url: "/auth"

function - Can execute functions at any point in a scenario:

- post:
    url: "/login"
    function: "doSomething"

Injecting Data from a Payload File

Artillery also lets you inject custom data through a payload file in CSV format. For example, instead of generating fake email addresses and passwords on the fly as we did in the previous section, you can have a predefined list of such data in a CSV file:

Dovie32@example.net,rwkWspKUKy
Allen.Fay@example.org,7BaFHbaWga
Jany30@example.org,CWvc6Bznnh
Dorris47@example.com,1vlT_02i6h
Imani.Spencer21@example.net,1N0PRraQU7

To access the data in this file, you need to reference it in the test script through the config.payload.path property. Secondly, you need to specify the names of the fields you'd like to access through config.payload.fields. The config.payload property provides several other options to configure its behavior, and it's also possible to specify multiple payload files in a single script.

config:
  target: "http://localhost:4000"
  phases:
    - duration: 60
      arrivalRate: 20
  payload:
    path: "./auth.csv"
    fields:
      - "email"
      - "password"

scenarios:
  - name: "Authenticating users"
    flow:
      - post:
          url: "/login"
          json:
            email: "{{ email }}"
            password: "{{ password }}"

Capturing Response Data From an Endpoint

Artillery makes it easy to capture the response of a request and reuse certain fields in a subsequent request. This is helpful if you're simulating flows with requests that depend on an earlier action's execution.

Let's assume you're providing a geocoding API that accepts the name of a place and returns its longitude and latitude in the following format:

{
  "longitude": -73.935242,
  "latitude": 40.730610
}

You can populate a CSV file with a list of cities:

Seattle
London
Paris
Monaco
Milan

Here's how you can configure Artillery to use each city's longitude and latitude values in another request. For example, you can use the values to retrieve the current weather through another endpoint:

config:
  target: "http://localhost:4000"
  phases:
    - duration: 60
      arrivalRate: 20
  payload:
    path: "./cities.csv"
    fields:
      - "city"

scenarios:
  - flow:
      - get:
          url: "/geocode?city={{ city }}"
          capture:
            - json: "$.longitude"
              as: "lon"
            - json: "$.latitude"
              as: "lat"
      - get:
          url: "/weather?lon={{ lon }}&lat={{ lat }}"

The capture property above is where all the magic happens. It's where you can access the JSON response of a request and store it in a variable to reuse in subsequent requests. The longitude and latitude properties from the /geocode response body (with the aliases lon and lat, respectively) are then passed on as query parameters to the /weather endpoint.

Using Artillery in a CI/CD Environment

An obvious place to run your load testing scripts is in a CI/CD pipeline so that your application is put through its paces before being deployed to production.

When using Artillery in such environments, it's necessary to set failure conditions that cause the program to exit with a non-zero code. Your deployment should abort if performance objectives are not met. Artillery provides support for this use case through its config.ensure property.

Here's an example that uses the ensure setting to assert that 99% of all requests have an aggregate response time of 150 milliseconds or less and that 1% or less of all requests are allowed to fail:

config:
  target: "https://example.com"
  phases:
    - duration: 60
      arrivalRate: 20
  ensure:
    p99: 150
    maxErrorRate: 1

Once you run the test, it will continue as before, except that assertions are verified at the end of the test and cause the program to exit with a non-zero exit code if requirements are not met. The reason for a test failure is printed at the bottom of the summary report.

All virtual users finished
Summary report @ 07:45:48(+0100) 2021-09-03
  Scenarios launched:  10
  Scenarios completed: 10
  Requests completed:  20
  Mean response/sec: 4
  Response time (msec):
    min: 1
    max: 487
    median: 2
    p95: 443.5
    p99: 487
  Scenario counts:
    0: 10 (100%)
  Codes:
    200: 20

ensure condition failed: ensure.p99 < 200

Aside from checking the aggregate latency, you can also run assertions on min, max, and median — the minimum, maximum, and median response times, respectively. Here's how to assert that requests never take more than 500 milliseconds to complete during a test run:

config:
  ensure:
    max: 500

The report for a failed test will indicate the reason for failure:

All virtual users finished
Summary report @ 08:29:59(+0100) 2021-09-03
  Scenarios launched:  10
  Scenarios completed: 10
  Requests completed:  20
  Mean response/sec: 3.64
  Response time (msec):
    min: 1
    max: 603
    median: 305.5
    p95: 602.5
    p99: 603
  Scenario counts:
    0: 10 (100%)
  Codes:
    200: 20

ensure condition failed: ensure.max < 500

Generating Status Reports in Artillery

Artillery prints a summary report for each test run to the standard output, but it's also possible to output detailed statistics for a test run into a JSON file by utilizing the --output flag:

$ artillery run config.yml --output test.json

Once the test completes, its report is placed in a test.json file in the current working directory. This JSON file can be visualized through Artillery's online report viewer or converted into an HTML report through the report subcommand:

$ artillery report --output report.html test.json
Report generated: report.html

You can open the report.html file in your browser to view a full report of the test run. It includes tables and several charts that should give you a good idea of how your application performed under load:

Extending Artillery With Plugins

Artillery's built-in tools for testing HTTP, Socket.io, and Websocket APIs can take you quite far in your load testing process. However, if you have additional requirements, you can search for plugins on NPM to extend Artillery's functionality.

Here are some official Artillery plugins that you might want to check out:

artillery-plugin-expect: Helps with adding expectations to HTTP requests for functional or acceptance testing.
artillery-plugin-publish-metrics: Used to send statistics from test runs to some external monitoring and observability systems.
artillery-plugin-fuzzer: Helps you fuzz test your APIs with random and unexpected payloads to your API endpoints so you can catch errors. It is based on the Big List Of Naughty Strings.
artillery-plugin-metrics-by-endpoint: Breaks down response time metrics by endpoint rather than displaying aggregate values across all endpoints.

You can also extend Artillery by creating your own plugins.

Use Artillery for Node.js Apps to Avoid Downtime

In this article, we've described how you can set up a load testing workflow for your Node.js applications with Artillery. This setup will ensure that your application performance stays predictable under various traffic conditions. You'll be able to account well for traffic-heavy periods and avoid downtime, even when faced with a sudden influx of users.

We've covered a sizeable chunk of what Artillery can do for you, but there's still lots more to discover.

Ensure you read the Artillery official documentation to learn about the other features on offer.

Thanks for reading, and happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Ayo is a Software Developer by trade. He enjoys writing about diverse technologies in web development, mainly in Go and JavaScript/TypeScript. You can learn more about him through his blog.

What's New in Node.js 17

Ayooluwa Isaiah — Wed, 17 Nov 2021 13:00:17 +0000

Node.js v17.0.0, the latest major version of the popular JavaScript runtime, has just been released. It supersedes v16 in the Current release line of the runtime. V16 is now in line to be promoted to the long-term support (LTS) channel on October 26, 2021, as it's an even-numbered release.

Despite being a relatively minor update, this release brings several refinements to the runtime, including more promisified APIs, JavaScript engine upgrades, and OpenSSL 3.0 support.

In this article, we'll take a look at some of the major highlights from this release and the implications for Node.js developers.

New Promise-based APIs

Node.js continues to promisify its core APIs as part of its strategic initiative plan. In the last few major Node.js releases, Promise-based APIs were added for the dns, fs, stream, and timers modules.

In Node.js 17, this ongoing promisification work has been extended to the readline module, primarily used for accepting input from the command line. The new APIs are accessible via the readline/promises module.

The old way of utilizing the readline module in Node.js v16 and earlier involved using callback functions, as shown below:

// main.mjs
import readline from "readline";
import process from "process";

const rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout,
});

rl.question(`What's your name?`, (name) => {
  console.log(`Hi ${name}!`);
  rl.close();
});

As of Node.js 17, you can now use await when importing from
readline/promises:

// main.mjs
import readline from "readline/promises";
import process from "process";

const rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout,
});

const name = await rl.question(`What's your name?`);
console.log(`Hi ${name}!`);
rl.close();

Stack Traces Now Include the Version of Node.js

When diagnosing a reported issue, a common question is: what version of Node.js executed the program?

Node.js 17 makes it easier to provide this information by including the version number at the end of a stack trace whenever an uncaught exception causes the process to exit:

file:///home/ayo/dev/demo/main.mjs:1
throw new Error("Uncaught exception");
      ^

Error: Uncaught exception
    at file:///home/ayo/dev/demo/main.mjs:1:7
    at ModuleJob.run (node:internal/modules/esm/module_job:185:25)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:281:24)
    at async loadESM (node:internal/process/esm_loader:88:5)
    at async handleMainPromise (node:internal/modules/run_main:65:12)

Node.js v17.0.0

If you'd rather omit this information in your program's stack trace, you can use the --no-extra-info-on-fatal-exception command-line flag when starting your Node.js scripts.

OpenSSL 3.0 Support

Node.js 17 now supports the newly unveiled OpenSSL 3.0 release.

The aim is for the APIs in OpenSSL 3.0 to be compatible with those provided in previous OpenSSL versions. However, stricter restrictions on the allowable key sizes and algorithms mean there might be some ecosystem impact — especially for users still using small keys or older algorithms.

This impact is reflected in the error message ERR_OSSL_EVP_UNSUPPORTED in Node.js 17 when your application or its dependencies uses an algorithm or key size that is not allowed in OpenSSL 3.0.

You can use the --openssl-legacy-provider command-line flag to enable the OpenSSL 3.0 legacy provider as a temporary way to ease these restrictions.

V8 Is Upgraded to v9.5

As of Node.js 17, the v8 JavaScript engine has been updated to v9.5. The changes in this release are primarily aimed at expanding internationalization for dates and calendars as well as for the output of time zones. It also implements the WebAssembly Exception Handling proposal, designed to reduce overhead compared to current JavaScript-based workarounds.

Deprecations and Removals

As a major release, Node.js 17 also comes with a few deprecations and removals.

A notable one is the deprecation of trailing slash pattern mappings which is unsupported in the import maps specification.

Upgrading to Node.js 17

You can download Node.js 17 to your computer using the appropriate link for your operating system and architecture provided on the Node.js download page. A better way to manage Node.js releases on your machine is to use a Node.js environment management tool like Volta, which allows you to install and switch between multiple Node.js versions seamlessly.

After installing the Volta CLI, run the command below to install the latest version of Node.js:

$ volta install node@latest
success: installed and set node@17.0.1 (with npm@8.1.0) as default

You can also install specific versions using the syntax below:

$ volta install node@lts # install latest lts version
$ volta install node@16.9.0 # install specific version 16.9.0
$ volta install node@12 # install the latest v12 release

When you install a Node.js release with Volta, it will co-exist with any other versions that you have already installed.

You can list the Node runtime versions in your toolchain using the command below:

$ volta list node
⚡️ Node runtimes in your toolchain:

  v14.8.0
  v14.17.5
  v16.7.0
  v16.8.0
  v16.9.0
  v17.0.1 (default)

Please see Volta's online documentation for more details on how it works and what it can do.

Wrapping up

You can examine the complete list of bug fixes, new features, and other changes included in this release in the official Node.js v17 release
notes.

To find out more about the Node.js project and how you can contribute, you can check out the list of Node.js open issues and Node.js contribution guidelines on the project's GitHub repository.

Thanks for reading, and happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Ayo is a Software Developer by trade. He enjoys writing about diverse technologies in web development, mainly in Go and JavaScript/TypeScript. You can learn more about him through his blog.

How to Dockerize an Existing Node.js Application

Ayooluwa Isaiah — Tue, 26 Oct 2021 12:42:45 +0000

Docker is a software platform that enables packaging an application into containers. These containers represent isolated environments that provide everything necessary to run the application. Dockerizing an application refers to packaging it in a Docker image to run in one or more containers.

Dockerizing an application involves specifying everything needed to run the application in a Dockerfile and then using the file to build a specialized Docker image that can be shared to multiple machines. A Docker image is a reproducible environment for the application that guarantees portability across machines.

In this tutorial, you'll learn the process of Dockerizing an existing Node.js application from scratch. We'll cover topics such as:

What the Dockerfile represents
Sharing Docker images to multiple machines
The basics of Docker Compose for orchestrating multi-container applications

After reading this article, you should be armed with enough knowledge to Dockerize your own applications, even if they're built with some other technology.

Setting Up a Demo Node.js Application

To demonstrate the concepts discussed in this article, we'll use a demo Node.js application that provides an endpoint for the retrieval of Covid-19 statistics. It uses the free API provided by disease.sh.

You can clone its GitHub repository to your computer using the command below:

$ git clone https://github.com/finallyayo/covid-node

Once downloaded, cd into the project folder and run yarn to install its dependencies. Afterward, open up the app.js file in your text editor. You should see the following content:

const fastify = require('fastify')({
  logger: true,
});
const got = require('got');
const NodeCache = require('node-cache');

const appCache = new NodeCache();

fastify.get('/covid', async function (req, res) {
  try {
    let covidAllStats = appCache.get('covidAllStats');

    if (covidAllStats == null) {
      const response = await got('https://disease.sh/v3/covid-19/all');
      covidAllStats = response.body;

      appCache.set('covidAllStats', covidAllStats, 600);
    }

    res
      .header('Content-Type', 'application/json; charset=utf-8')
      .send(covidAllStats);
  } catch (err) {
    fastify.log.error(err);
    res.code(error.response.code).send(err.response.body);
  }
});

fastify.listen(4000, '0.0.0.0', (err, address) => {
  if (err) {
    fastify.log.error(err);
    process.exit(1);
  }

  fastify.log.info(`server listening on ${address}`);
});

This application provides a single endpoint (/covid) that returns the aggregated global Covid-19 totals to date. Once retrieved from the API, the data is subsequently cached in-memory for 10 minutes.

Specifying '0.0.0.0' as the address is essential when deploying to Docker because Docker containers do not default to exposing mapped ports to localhost. If this address is missing, your application might be inaccessible despite starting successfully in the container.

Go ahead and start the server with yarn dev, then make a GET request to the /covid endpoint with curl or some other tool. You should see a JSON response similar to the output shown below:

$ curl http://localhost:4000/covid
"updated":1629986413872,"cases":214977601,"todayCases":270792,"deaths":4481152,"todayDeaths":5588,"recovered":192301169,"todayRecovered":273952,"active":18195280,"critical":112761,"casesPerOneMillion":27580,"deathsPerOneMillion":574.9,"tests":3264569720,"testsPerOneMillion":416082.42,"population":7845968850,"oneCasePerPeople":0,"oneDeathPerPeople":0,"oneTestPerPeople":0,"activePerOneMillion":2319.06,"recoveredPerOneMillion":24509.55,"criticalPerOneMillion":14.37,"affectedCountries":223}⏎

Although this is a very simple application, it will suffice to demonstrate the concepts of Docker covered in this tutorial.

In the next section, we'll take a look at how to set up the Docker Engine locally on your machine.

Installing Docker

Before you can Dockerize an application, you need to install the Docker Engine. The official Docker manual provides a guide for installing the software on a variety of operating systems, most notably on macOS, Windows, and a variety of Linux distributions. Ensure you install the latest stable release — v20.10.x at the time of writing.

$ docker -v
Docker version 20.10.5, build 55c4c88

Setting Up a Dockerfile

Once the Docker Engine has been installed, the next step is to set up a Dockerfile to build a Docker image for your application. An image represents an immutable snapshot of an environment that contains all the source code, dependencies, and other files needed for an application to run. Once a Docker image is created, it can be transported to another machine and executed there without compatibility issues.

Docker images are assembled through a Dockerfile. It is a text file that contains a set of instructions executed in succession. These instructions are executed on a parent image, and each step in the file contributes to creating an entirely custom image for your application.

Let's go ahead and create a Dockerfile for our demo application at the root of the project directory:

$ touch Dockerfile

Open up the Dockerfile in your text editor and add the following line to the file:

FROM node:16-alpine

The above specifies the base image to be the official Node.js Alpine Linux image. Alpine Linux is used here due to its small size, which helps a lot when transporting images from one machine to the other.

The next line in the Dockerfile is shown below:

WORKDIR /app

The WORKDIR instruction sets the working directory to /app. This directory will be created if it doesn't exist.

Use the following lines to install your application's dependencies: a crucial step for building your Docker image. Note that the lines that start with # denote a comment.

# Copy and download dependencies
COPY package.json yarn.lock ./
RUN yarn --frozen-lockfile

# Copy the source files into the image
COPY . .

Next, we need to expose the port that the application will run on through the EXPOSE instruction:

EXPOSE 4000

Finally, specify the command for starting the application:

CMD yarn start

You can see the entire Dockerfile below:

FROM node:16-alpine
WORKDIR /app
COPY package.json yarn.lock ./
RUN yarn --frozen-lockfile
COPY . .
EXPOSE 4000
CMD yarn start

Build the Docker Image

Now that the Dockerfile is complete, it's time to build the Docker image according to the instructions in the file. This is achieved through the docker build command. You need to pass in the directory where the Dockerfile exists and your preferred name for the image:

$ docker build . -t covid

If all goes well and the build succeeds, you will see the messages below at the end of the command's output:

Successfully built 973edfcb25d2
Successfully tagged covid:latest

You can run docker images to view some basic info about the created image:

$ docker images
REPOSITORY      TAG         IMAGE ID       CREATED         SIZE
covid           latest      973edfcb25d2   2 minutes ago   137MB

Run the Docker Image in a Container

Use the docker run command to run your newly minted Docker image inside of a container. Since the application has been built into the image, it has everything it needs to work. It can be launched directly in an isolated process. Before you can access your running image inside the container, you must expose its port to the outside world through the --publish or -p flag.

This lets you bind the port in the container to a port outside the container.

$ docker run -p 4000:4000 covid

The command above starts the covid image inside of a container and exposes port 4000 inside the container to port 4000 outside the container. You can subsequently access the routes on your server through http://localhost:4000.

Sharing Docker Images

You can transfer Docker images from one machine to the other in a variety of ways. The most popular method involves using the docker push command to push the image to the official Docker registry and retrieving it through the docker pull command.

You need to sign up for a free account at Docker Hub first. After the signup process is complete, head over to the Repositories page, and create a new repository. Give it a name and set its visibility to "Public" or "Private".

Note that free accounts have access to a limited number of private repos.

Once you've created a repository, enter the docker login command on your terminal to log in to Docker Hub on your machine.

Before you push the image to Docker Hub, you need to update the image tag to match your repository namespace: <your docker username>/<repo name>. This is because the docker push command expects an argument in this format.

Enter the command below to tag your covid image with a new name. Ensure you replace <your docker username> with your actual docker username.

$ docker tag covid <your docker username>/covid

Finally, push the image to Docker Hub using the docker push command, as shown below:

$ docker push <your docker username>/covid

Once the image is pushed successfully to the registry, it will be reflected in your repository dashboard:

You can pull the image on any machine with docker installed through the command below. If the repository is private, you'll need to log in first through the docker login command. Keep in mind that the speed of downloading an image from the registry depends on the image size and the speed of your internet connection. This is one of the reasons why smaller Docker images are preferred in general.

$ docker pull <your docker username>/covid

Note that you can also choose to share Docker images through registries provided by other cloud services such as GitLab, Google Cloud, RedHat, and others. You can even set up your own private registry on a dedicated server for use within an organization.

Share Docker Images without Using a Registry

An alternative way to share a Docker image with others is to export it as a .tar file and transfer it to a different machine through any preferred transfer method. This helps you transfer the Docker images between machines in cases when using a Docker registry is not desirable or possible, for whatever reason. The docker save command is what you need to use for exporting a Docker image:

$ docker save covid > covid.tar

The above command will export the covid image to a covid.tar file in the current directory. This file may then be transferred to a remote machine and loaded into the machine's local registry through the docker load command:

$ docker load < covid.tar
Loaded image: covid:latest

Deploy Your Dockerized Node.js Application to Production

The easiest way to deploy a Dockerized application on a remote server is to transfer the application's image with docker pull and then use docker run.

This runs the application in a container similar to how you'd do it in your development environment. However, such a strategy is suboptimal for a truly production-ready application.

Unlike our demo application, a real-world product will likely be composed of several different services that depend on each other for the application as a whole to properly work. Deploying to production usually means starting all the component services in the right order to ensure a smooth operation. You also need a strategy for other tasks, such as restarting a service in case of failures, aggregating logs, and performing health checks. All these concerns — and more — can be handled through Docker Compose.

Docker Compose coordinates multi-container Docker applications through a single command. It relies on a Compose file that provides a set of instructions to configure all the containers that should be spawned. Here's what the Compose file (docker-compose.yml) for our demo application looks like:

version: '3'
services:
  web:
    image: covid
    ports:
      - "4000:4000"
    environment:
      NODE_ENV: production

The above Compose file uses version 3 of the Compose file format and defines a single service called web that uses the covid image we previously set up. If you leave out the image property, a Docker image from the Dockerfile will be built in the current directory and used for the service. The ports property defines the exposed ports for the container and host machine, and the environment property sets up any necessary environmental variables.

Once you have a docker-compose.yml file, you can start the defined services with the docker-compose up command. Make sure you have docker-compose installed before running the command, otherwise, find out how to install Docker Compose on your operating system.

$ docker-compose up
Recreating covid-node_web_1 ... done
Attaching to covid-node_web_1
web_1  | yarn run v1.22.5
web_1  | $ node app.js
web_1  | {"level":30,"time":1630001521702,"pid":28,"hostname":"204c8ce51d52","msg":"Server listening at http://0.0.0.0:4000"}

This command will launch the containers for the defined services, and they will be accessible on the specified ports. Note that if you exit this command (such as by pressing Ctrl-C), every spawned container will stop immediately. To prevent this from happening, append the --detach flag so that the containers start in the background and keep running.

$ docker-compose up --detach

We've only scratched the surface of the workflows a Compose file can achieve. Ensure to check out the full documentation to learn more about all the available options. The docker-compose CLI also provides several other important commands you should know about to get the most out of it. You can examine each of them through the --help flag or the CLI reference page.

Wrap-up and Further Docker Reading

In this article, we covered the process of Dockerizing an existing Node.js application, building containers, and deploying to production through Docker Compose.

Keep in mind that there's a lot more to Docker than can be covered in one article. Refer to the official documentation to learn more about best practices for writing a Dockerfile, securing a Docker container, logging, and other important topics to use Docker effectively in your application workflow.

Thanks for reading, and happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Ayo is a Software Developer by trade. He enjoys writing about diverse technologies in web development, mainly in Go and JavaScript/TypeScript. You can learn more about him through his blog.

Best Practices for Logging in Node.js

Ayooluwa Isaiah — Wed, 08 Sep 2021 12:08:33 +0000

This post was updated on 27 April 2023 to include information about logging with Winston and AppSignal.

Good logging practices are crucial for monitoring and troubleshooting your Node.js servers. They help you track errors in the application, discover performance optimization opportunities, and carry out different kinds of analysis on the system (such as in the case of outages or security issues) to make critical product decisions.

Even though logging is an essential aspect of building robust web applications, it's often ignored or glossed over in discussions about development best practices. Knowing what and how to log is tricky because it's often difficult to understand what information you’ll need during troubleshooting.

Since we understand how essential logging is, we need to practice good logging habits. We should accumulate logs that make it easier to diagnose the root cause of problems and solve potential issues before they impact end-users.

This article will outline some best practices to follow when writing logs in a Node.js application.

1. Use a Node.js Logging Library

Node.js developers tend to rely on the runtime's console methods (such as console.log()) to log events since it is built into the runtime, and offers a familiar API similar to the JavaScript console mechanism provided by web browsers.

While console.log() has its uses, it is not an adequate solution for implementing logging in a production application. It lacks support for features and configuration options that are considered essential to a good logging setup. For example, console methods do not support log levels like warn, error or debug, despite the provision of methods like console.warn(), console.error(), and console.debug(), respectively. Those are simply functions that print to the standard output or standard error without indicating log severity.

A good logging library provides a robust feature set that makes it easier to centralize, format, and distribute the logs to fit your needs. For example, a typical logging framework will provide various options for where to output log data (such as the terminal, filesystem, or a database), while also supporting the ability to send the logs over HTTP if you want to transport your log entries to a log management service.

There are three major concerns for choosing a suitable logging library: recording, formatting, and storing messages. You need to make sure that your library of choice addresses all three concerns in a satisfactory manner. Another critical consideration for selecting a logging library is performance. Since the logger will be used a lot throughout the codebase, it can harm your application's runtime performance. Therefore, you should also investigate the performance characteristics of a library, and see how it compares to alternatives.

In the Node.js ecosystem, there are several popular options to consider. Most of them offer similar features, but they also have their differences — so it's vital that you try them out yourself and see which one caters to your use case the best:

Winston — The most popular logging library, with support for multiple transports. This allows you to easily configure your preferred storage location for your logs.
Pino — Pino's biggest attraction is its speed. It claims to be up to five times faster than alternatives, in many cases.
Bunyan — Another feature-rich logging framework that outputs in JSON by default and provides a CLI tool for viewing your logs.
Roarr — Roarr is a different kind of logger that works in Node.js and the browser.

For the rest of the article, we'll use winston to demonstrate the different concepts. This is not necessarily an endorsement of winston; it's only being used here because it's the most popular logging framework for Node.js. You can follow along by installing winston in your project though npm:

npm install winston

2. Use the Correct Log Levels

If you've spent a reasonable amount of time in the software industry, especially supporting production applications, you may have observed the use of log levels, at least in passing. They provide a way to differentiate between the types of events in a system and add context to how important each event is. If you correctly utilize log levels in your application, it will be easy to distinguish between critical events that need to be immediately addressed versus purely informative events.

Although logging systems give different names to severity levels, the concepts remain largely the same. Here are the most common log levels that you are likely to encounter, regardless of the logging framework you choose (in decreasing order of severity):

FATAL: Used to represent a catastrophic situation — your application cannot recover. Logging at this level usually signifies the end of the program.
ERROR: Represents an error condition in the system that happens to halt a specific operation, but not the overall system. You can log at this level when a third-party API is returning errors.
WARN: Indicates runtime conditions that are undesirable or unusual, but not necessarily errors. An example could be using a backup data source when the primary source is unavailable.
INFO: Info messages are purely informative. Events that are user-driven or application-specific may be logged at this level. A common use of this level is to log interesting runtime events, such as the startup or shutdown of a service.
DEBUG: Used to represent diagnostic information that may be needed for troubleshooting.
TRACE: Captures every possible detail about an application's behavior during development.

The winston library uses the following log levels by default — with error being the most severe and silly being the least:

{
  error: 0,
  warn: 1,
  info: 2,
  http: 3,
  verbose: 4,
  debug: 5,
  silly: 6
}

If the defaults do not suit your needs, you can change them while initializing a custom logger. For example, you can instead use the log levels discussed above.

const { createLogger, format, transports } = require('winston');

const logLevels = {
  fatal: 0,
  error: 1,
  warn: 2,
  info: 3,
  debug: 4,
  trace: 5,
};

const logger = createLogger({
  levels: logLevels,
  transports: [new transports.Console()],
});

When you want to log a message, you can reference the desired level directly on the custom logger, as shown below:

logger.info('System launch'); // {"message":"System launch","level":"info"}
logger.fatal('A critical failure!'); // {"message":"A critical failure!","level":"fatal"}

Winston also allows you to define a level property on the logger and on each transport which specifies the maximum level of messages that should be logged. For example, you can run your programs at info level by default, then switch to debug or trace level when you need to troubleshoot an issue or deploy to a testing environment. You should control this setting through an environmental variable.

const logger = winston.createLogger({
  level: 'warn',
  levels: logLevels,
  transports: [
    new transports.Console({ level: 'info' }),
  ]
});

3. Use Structured Logging

When defining how your log messages look, the priority should be to make your log entries easy to read for both humans and machines.

One of the primary goals of logging is to enable post-mortem debugging, and this will involve reading log entries to reconstruct the steps that led to an event in the system. Having human-readable log entries will make this thankless task much easier for developers and sysadmins. It's also important to use a structured format that is easy to parse by machines. This allows for some automated processing on the logs (such as for alerting or auditing purposes).

JSON is a universal favorite for structured log entries because it is ubiquitous and easily readable by humans. It is also highly machine-readable and easily converted to other formats, even when working with other programming languages.

When logging in JSON, it's necessary to use a standard schema so that the semantics of each field is clearly defined. This also makes it easy to find what you're looking for when analyzing log entries.

Winston outputs a JSON string by default with two fields: message and level. The former contains the text that is being logged, while the latter signifies the log level. Customizing the output is straightforward through winston.format, which uses logform to implement its different formats. For example, if you wanted to add a timestamp to each log entry, you can do so by combining the timestamp and json formats as shown below:

const { createLogger, format, transports } = require('winston');

const logger = createLogger({
  format: format.combine(format.timestamp(), format.json()),
  transports: [new transports.Console({})],
});

This will produce log entries in the following format:

{"message":"Connected to DB!","level":"info","timestamp":"2021-07-28T22:35:27.758Z"}
{"message":"Payment received","level":"info","timestamp":"2021-07-28T22:45:27.758Z"

Ensure you read the logform docs to learn about the predefined formats and how to create your own custom formats.

4. Write Descriptive Messages

Log entries should adequately describe the events that they represent. Each message should be unique to the situation and should clearly explain the event that occurred at that point. In the event of an emergency, your log entries may be the only source of information to help you understand what happened, so it's important to get this aspect of logging right!

Here's an example of an inadequate log entry being used to communicate a request failure:

Request failed, will retry.

The message above doesn't provide any insights into:

The specific request that failed
The reason for its failure
The length of time before the request is retried

We can probably find out the answers to some of these questions by looking at other places such as other log entries or even the code itself. However, it is better to make the log entry more valuable on its own, through a more descriptive message:

"POST" request to "https://example.com/api" failed. Response code: "429", response message: "too many requests". Retrying after "60" seconds.

This second message is much better because it provides adequate information about the failed request, including the status code and response message, and also notes that the request will be retried in 60 seconds. If all your messages are just as descriptive, you'll have a more pleasant time when
attempting to make sense of your logs. Other examples of good log messages include the following:

Status of task id "1234" changed from "IN_PROGRESS" to "COMPLETED".
SomeMethod() processed "100" records in "35ms".
User registration failed: field "email" is not valid email address; field "password" is below the minimum 8 characters.

When composing log messages, include all the relevant details pertaining to the event without being unnecessarily verbose. This will prevent other log readers (which could include your future self) from being overwhelmed with too much information to sift through. Your log messages should also be able to stand on their own; don't rely on a previous message's content to provide the context for a later entry.

5. Add the Right Amount of Context to Your Logs

Besides writing a descriptive log message, you also need to include the right amount of context in the log entry. Context makes it possible to quickly reconstruct the actions leading up to an event. Add basic information to the log, such as the timestamp of the event and the method where it occurred (or a stack trace, in the case of errors). You should also add data points relevant to the flow of the operation that triggered the event. These data points may be generated at different parts of the operation flow and aggregated at the point of logging.

In the context of a billing service, log entries generated by the system could include several data points, including:

Session identifiers
Usernames and IDs
Product or transaction identifiers
The current page that the user is on

You can use each of the above data points to track a user's flow through an entire checkout process. If an important event occurs, the available data will be automatically appended to the log output, and it will be possible to identify:

the circumstances that led to the event (such as the user who experienced the event)
the page where it occurred
the transaction and
product id that triggered the event.

These data points also let you filter log entries based on a common identifier such as the user id or product id.

Winston provides the ability to add global metadata (such as the component or service where an event occurred) to every generated log entry. In a complex application, this information in your logs is helpful for troubleshooting issues because it immediately directs you to the point of failure.

You can configure this when creating the logger for the component or service:

const logger = createLogger({
  format: format.combine(format.timestamp(), format.json()),
  defaultMeta: {
    service: 'billing-service',
  },
  transports: [new transports.Console({})],
});

A service field will be included in all logs created by the logger object:

{"message":"Order \"1234\" was processed successfully","level":"info","service":"billing-service","timestamp":"2021-07-29T10:56:14.651Z"}

To add metadata to individual entries, you need to create a context or metadata object that you can pass around throughout the flow of an operation so that the data is accessible at logging points. You can also utilize the concept of child loggers to add metadata at logging points:

const ctx = {
  userId: '090121',
  productId: 'creme-de-la-creme',
};

logger.child({ context: ctx }).info('Order "1234" was processed successfully');
// {"context":{"userId":"090121","productId":"creme-de-la-creme"},"message":"Order \"1234\" was processed successfully","level":"info","service":"billing-service","timestamp":"2021-07-29T12:20:13.249Z"}

6. Avoid Logging Sensitive Information

Regardless of whether you're in an industry with strict rules around compliance (such as healthcare or finance) or not, it's important to avoid including sensitive information in your logs.

Sensitive information includes social security numbers, addresses, passwords, credit card details, access tokens, and similar data types. Since log messages are often stored in plain text, such data will be exposed if the logs fall into the wrong hands. You also have to ensure you are not falling
afoul of regulations that apply to countries where your product is operational (such as GDPR) by recording certain pieces of information.

You can avoid accidentally leaking sensitive data in your logs by minimizing which parts of the system work with that data. For instance, credit card details should only be seen by the billing component of your system, and sensitive data should be kept out of URLs — redacted where possible.

Although this isn't a foolproof solution, you can also use a blocklist to prevent specific fields from ever making it into the logs.

7. Log for Auditing and Profiling Reasons

We primarily use logs to diagnose issues and find the root cause of bugs. However, logs can also prove invaluable when auditing or profiling a system, or perhaps to generate interesting statistics about system behavior.

For example, you can log details of what users are doing on the system (like user sign-ins, the resources they created or accessed, etc.). In the absence of specialized tools, you can also use logging to profile your code by reporting how long an operation took or how many times a function was executed. The
insights gleaned from this data can help you improve the performance of your application's hotspots.

Winston provides a simple profiling mechanism that you can take advantage of:

logger.profile('test', { level: 'info' });

// Returns an object corresponding to a specific timing
const profiler = logger.startTimer();

setTimeout(() => {
  // End the timer and log the duration
  profiler.done({ message: 'Logging message' });
}, 1000);

A durationMs field will be included in the output:

{"message":"Logging message","level":"info","durationMs":1001,"timestamp":"2021-07-29T14:17:55.097Z"}

8. Automatically Log Uncaught Exceptions and Unhandled Promise Rejections

When you encounter an uncaught exception or unhandled promise rejection, it is always considered good practice to crash the program. Use a process manager like PM2 to automatically restart the process and restore the program to a clean state.

To understand why such an event has occurred, it's also necessary to log the details of the exception or promise rejection before exiting. Winston provides handlers for both situations, which may be configured on a logger instance:

const logger = createLogger({
  transports: [
    new transports.File({ filename: 'file.log' })
  ],
  exceptionHandlers: [
    new transports.File({ filename: 'exceptions.log' })
  ],
  rejectionHandlers: [
    new transports.File({ filename: 'rejections.log' })
  ]
});

In the example above, uncaught exceptions will be logged into an exceptions.log file, while unhandled rejections will be logged into a rejections.log file. Entries will automatically include the full stack trace as well as information about process arguments and memory usage associated with the exception, giving you all the details you need to find the root of the problem.

Learn how to configure Winston's unhandled exception handler.

9. Centralize and Monitor Your Logs with Winston and AppSignal

Managing a Node.js application often involves handling vast amounts of logs generated by the system. As your application grows, these logs can become increasingly difficult to track and analyze, making it crucial to centralize them in one unified location. By consolidating logs from various sources, you can streamline the monitoring and troubleshooting process, providing better visibility into your application's performance and ensuring a more reliable user experience.

Centralizing logs not only simplifies management, but also offers several key benefits:

Easier log analysis: Combining logs from multiple instances and sources allows for more efficient and comprehensive analysis, making identify patterns, trends, and potential issues.
Faster troubleshooting: When issues arise, centralized logs make it easier to pinpoint the root cause, reducing the time spent on debugging and improving the speed at which you can resolve problems.
Improved performance monitoring: With all logs in one place, you can easily monitor the overall health of your application, enabling you to identify performance bottlenecks and optimize your code accordingly.
Alerting and notifications: You can set up alerts based on specific conditions, ensuring that you're immediately notified of critical issues.
Regulatory compliance: For organizations subject to regulatory requirements, centralizing logs can help meet compliance standards by providing a clear audit trail and ensuring data integrity.

Winston can be configured to integrate with a log management platform like AppSignal to provide all the above benefits and more. All you need to do is set up the AppSignal transport for Winston as shown below:

const { createLogger, transports } = require("winston");
const { WinstonTransport } = require("@appsignal/nodejs");

const logger = createLogger({
  transports: [new WinstonTransport({ group: "nodejs_app" })],
});

logger.info("Hello from Winston Logger!");

You'll need to install the AppSignal SDK for Node.js,
create a log source via the AppSignal dashboard, and initialize the AppSignal client instance with the provided API Key. Once you're all set up, your application logs will be streamed directly to AppSignal.

Here's an example of how your logs might look in AppSignal:

Learn more about AppSignal's logging integrations for Node.js.

Node.js Logging Tips: Next Steps

In this article, we've covered some invaluable practical tips to help you write more useful log entries. Next, you should add logging to your Node.js server and configure it while keeping the above best practices in mind.

If you have any additional tips regarding logging in Node.js that were not discussed in this post, feel free to share them with me on Twitter.

Thanks for reading, and happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Ayo is a Software Developer by trade. He enjoys writing about diverse technologies in web development, mainly in Go and JavaScript/TypeScript. You can learn more about him through his blog.

Migrating away from Google Analytics

Ayooluwa Isaiah — Mon, 13 Jul 2020 16:44:00 +0000

Google analytics has been my go-to analytics tool ever since I became a web developer a few years ago. It's the de-facto standard for website usage tracking and is free to use, feature-rich and easily accessible on web and mobile. However, over the last couple of months, I contemplated moving away from the service for some reasons.

My first gripe with Google Analytics is that the interface is quite slow and confusing to use. It's a busy interface and I always have to fumble around to find the relevant metric I'm looking for. The mobile app is much better in this regard at least in terms of speed.

I also realised that I wasn't even using up to 10% of the features provided by the service. For the most part, all I cared about is the number of visitors, page views, and where visitors are coming from (search engines, social media e.t.c.) so all the other metrics that Google collects are not important for me.

Due to this reason, I felt it wasn't an acceptable trade-off to load the 19 KB tracking script just to view basic website statistics. When decompressed, the analytics script is about 46 KB which is just extra work for the browser to parse and execute and for not a lot of gain in the context of my needs.

Another major issue is privacy. Google Analytics is free to use (for up to 10 million hits per month), but it's only because the company recoups the cost of running the service by using the aggregated data to track users for the purpose of showing personalised ads.

As a result, the analytics script is often blocked by ad blockers and other anti tracking extensions including those built into browsers such as Firefox, Brave, and Vivaldi meaning that the reported statistics aren't all that accurate especially for sites with tech-savvy audiences like mine.

Evaluating alternatives

Quite a few Google Analytics alternatives have emerged recently with many beginning to recognise that Google has way too much influence on the web and also because of the usability issues discussed above.

My main criteria for a replacement was that it should be privacy-respecting, lightweight and easy to use with a good balance of features. I found some discussions on Hacker News on alternatives to Google Analytics and proceeded to check out some of the suggestions.

Goat Counter was the first option I looked into. It ticks most of the boxes and is fully open-source, easy to self-host and has a sane pricing model but I was not in love with the user interface of the metrics dashboard. Another option was Simple Analytics, but it's proprietary and too expensive for me at $19/month for ~100k views monthly.

Eventually, it came down to a choice between Plausible and Fathom. They both have a similar feature set, a good user interface to view my stats, and lightweight tracking scripts. I decided to go with Plausible in the end.

It mostly came down to pricing (Plausible starts at $12/month for ~100k views with a 33% discount if billed annually), and the fact that the codebase is fully open source and supports
self-hosting without restrictions. On the other hand, Fathom starts at $14/month for the same amount of views and the V2 of the service is no longer open source. They have a Lite version on GitHub, but it's clear from the Readme file that they don't want anyone to use it.

Switching to Plausible

I decided to take advantage of Plausible's generous 30-day trial to try out the service and get a feel for how it works. Their tracking script is one of the lightest out there coming in at around 834 bytes (1.3 KB uncompressed) which is more than 20 times lighter than GA's tracking script.

After adding the script to my site, the analytics began to show up almost immediately. The dashboard interface is quite polished and supports all of the basic metrics such as page views and unique visitors, browser and operating system statistics, device screen sizes, locations, and referrer information.

It also supports event tracking, weekly or monthly email reports, and the ability to proxy the tracking script through a custom domain which can be quite handy because some browsers and extensions block all analytics services including privacy-friendly ones.

I decided to run Plausible side by side with Google Analytics for a few days just to see how the reported stats compare. Unsurprisingly, Plausible's reporting for page views was consistently about 30-40% higher presumably because GA is blocked by some clients. GA is gone for good now though.

Wrap up

The experience of switching away from Google Analytics to Plausible was painless and I'm really happy with how it turned out so far. I think both Plausible and Goat Counter are excellent choices for anyone looking for a privacy friendly alternative to Google Analytics. The former just has a bit more visual polish while the latter is significantly easier to self-host but you really can't go wrong with either.

Thanks for reading!

A guide to setting up Vim for JavaScript development

Ayooluwa Isaiah — Mon, 06 May 2019 00:00:00 +0000

I’ve been using Vim for close to two years now, and do all my JavaScript development work in it. In the process, I’ve tweaked my configuration several times until I finally achieved a setup that I can say I’m really happy with. So, in this article, I want to talk about some of the plugins and tools that I’ve taken advantage of to make writing JavaScript a more pleasurable experience in Vim.

All the plugins and settings were tested, and confirmed to work on Vim 8.1 and Neovim 0.4.0, the latest versions at the time of writing. As this post mainly deals with installing and configuring plugins for Vim, I recommend that you adopt vim-plug as your plugin manager if you don’t use one already.

Syntax highlighting

Vim supports basic syntax highlighting for JavaScript but I found it suboptimal especially when it comes to modern ES2015+ syntax, and it doesn’t support JSX when working with React. I found that vim-javascript and vim-jsx solved my problems in both instances.

However, I later replaced those two with vim-polyglot which is a plugin that bundles several other syntax plugins for over 100 languages, and loads them on demand so that performance is not affected.

So, instead of having to add another plugin to provide syntax highlighting for .vue files for example, vim-polyglot takes care of that without much fuss. This means I’m covered when switching between different JavaScript frameworks as well as other programming languages.

Linting your code with ESLint

Code linting helps you identify potential errors in your code without having to run the code. It also helps you avoid problematic patterns or code that doesn’t adhere to certain style guidelines. ESLint has become the dominant linting tool for JavaScript developers due to it’s comprehensive support for all the modern features of the language, and also its easy extensibility.

If you’re not using ESLint already, follow this guide to set it up in your JavaScript project. In order for ESLint to work, you need to configure it with specific rules. You can use the recommended configuration which only addresses common problems, or check out Airbnb’s style guide if you want a more comprehensive set of rules.

While ESLint was built to be used via the command line, many people (myself included) prefer to integrate it into their code editor to get live feedback on code as it is being written. Bringing ESLint integration into Vim is pretty easy with ALE.

Once you install ALE, open up Vim in a JavaScript project that has been configured to use ESLint, and try to create an error on purpose. It should work straightaway without further configuration.

As you can see, indicators for errors and warnings appear next to the problematic lines as the code is written, and the reason for the error is printed at the bottom of the screen when the cursor is on the line where is error is found. In this case, ESLint is warning me that about unused variables and also about the console statement.

Additionally, you can open the location list to view all the errors in a file and jump to each error using :lnext (or :lne) to navigate to the next item and :lprevious (or :lpr) to navigate to the previous item.

An easier way to jump between linting errors quickly is to use the <Plug> keybindings shown below. You can change [c and ]c to your preferred key combination.

nmap <silent> [c <Plug>(ale_previous_wrap)
nmap <silent> ]c <Plug>(ale_next_wrap)

By the way, I’m not a fan of the default indicators used by ALE for errors and warnings, so I changed them to something that’s aesthetically more pleasing using the following configuration:

let g:ale_sign_error = '❌'
let g:ale_sign_warning = '⚠️'

Code formatting with Prettier

Prettier is a code formatting tool that has gained huge momentum in the JavaScript ecosystem since coming on the scene around early 2017. It basically ensures that a codebase complies to a set code style guide regardless of the number of developers working on the project.

It works by formatting a file (or many files) according to the options specified in its configuration file. As such, all files formatted by Prettier adhere to the same guidelines for indendation, quote style, spacing, line width, and several other stylistic concerns.

To use Prettier, you need to add it to your project. You can use Prettier’s CLI, or configure it as a pre-commit hook, but I prefer the instant feedback of having it format my code immediately I save a file.

ESLint also has some code fixing capabilities, but some of these conflict with Prettier’s methods. It is better to have ESLint take care of code quality concerns, while leaving Prettier to focus on code formatting concerns.

To achieve this, you need to disable the ESLint rules that conflict with Prettier with eslint-config-prettier while adding Prettier’s code fixing capabilities to ESLint using eslint-plugin-prettier. To do so, run the following command in your project directory:

$ yarn add eslint-config-prettier eslint-plugin-prettier -D

Then add the following to your ESLint configuration file. This configures both eslint-plugin-prettier and eslint-config-prettier in a single step.

{
  "extends": ["plugin:prettier/recommended"]
}

That’s all you need to do to integrate Prettier with ESLint. The next step is to configure Vim so that all JavaScript code is formatted nicely as soon as a file is saved. ALE solves this nicely for us!

In addition to supporting linters that report errors, ALE can also run fixers to format the code in a Vim buffer. All you need to do is specify eslint as a fixer for JavaScript in your .vimrc as shown below:

let g:ale_fixers['javascript'] = ['eslint']

" Fix files automatically on save
let g:ale_fix_on_save = 1

At this point, linting and formatting errors will be annotated by ESLint, while Prettier formats the code on save.

You can also format the code on demand using the :ALEFix command. The easiest way to use it is to create a key mapping in your Vim configuration file.

nmap <F6> <Plug>(ale_fix)

Super fast fuzzy file search

File searching is important in any editor, and while Vim supports this via the :find command, it’s very basic and doesn’t support fuzzy finding as far as I can tell. There are a couple of tools out there that bring fuzzy file search to vim. After using CtrlP for a while, I switched to Fzf.vim which is a wrapper for the command line fuzzy finder with the same name, Fzf.

Fzf looks like a simple tool on the surface, but it’s actually very powerful. Here’s a few ways I take advantage of it on a daily basis:

Fuzzy file finder

My primary use of Fzf is to search files in a project. This is achieved using the :Files command which I heartily bound to Ctrl-P in my Vim config. I also modified the $FZF_DEFAULT_COMMAND variable to use Ripgrep for listing the files. After installing Ripgrep, put this in your .bashrc or .zshrc to search for all files except those in hidden directorys (like .git) or those that have been ignored in your .gitignore file:

export FZF_DEFAULT_COMMAND='rg --files --follow --hidden'

# Fish syntax
# set -gx FZF_DEFAULT_COMMAND 'rg --files --follow --hidden'

Additonally, Fzf provides the :Buffers amd :History commands for searching open buffers and your buffer history. I have mapped these to <Leader>b and <Leader>h respectively.

nnoremap <C-p> :Files<CR>
nnoremap <Leader>b :Buffers<CR>
nnoremap <Leader>h :History<CR>

Similarly to CtrlP, once Fzf is open, you can hit the Enter key to open the selected file in the current window, or use Ctrl-T, Ctrl-X or Ctrl-V to open selected files in a in new tab, horizontal split, or vertical split respectively.

Searching for tags

Fzf provides two main commands for searching tags in a project. The first one:BTags allows you to search for tags in the current buffer. It doesn’t require a tags file, and is great for jumping between methods quickly.

The second one, :Tags, allows you to perform project wide search for tags, but it does require a tags file. You can use Universal Ctags to generate a tags file for your project and combine it with Gutentags to automate the creation and updating of tags in the background.

I have mapped both commands mapped to <Leader>t and <Leader>T respectively.

nnoremap <Leader>t :BTags<CR>
nnoremap <Leader>T :Tags<CR>

Project-wide search

The :Rg command provided by Fzf uses Ripgrep to find all occurrences of a pattern across an entire project, so you can search for a specific string, select the matches and navigate the results using the quickfix list. I often use it to check for where a method is used in my codebase, and I found it incredibly helpful in a recent project I was refactoring.

When using :Rg, you can narrow down the results and hit Enter on the selected result to open the file in the current window. This will skip the quickfix list altogether. To select multiple results, just use theTab key on each file you want to select or Alt-A to select all results and then Enter to populate the quickfix list which you can navigate with :cnext and :cprev.

There is so much more that can be done with Fzf. For example you can navigate Vim’s :help documentation with :Helptags, search for Snippets (:Snippets), Git Commits (:Commits), Command history (:History:), and more. Use :h fzf-vim-commands to find out all the options available to you.

Intelligent code completion

For most people, having a good code completion experience is pivotal to having a modern development setup. Vim supports some basic functionality out of the box through omnicomplete, but we’re going to replace that with something much more powerful.

Since I started using Vim, I’ve tried an few plugins for auto-completion, including Deoplete and YouCompleteMe but neither provided the experience I desired.

I discovered Coc.nvim recently and it’s been great! It brings several IDE-like features to Vim and is easy to set up. It’s built upon the concept of language servers, which power features like auto-completion, go-to-definition, hover tooltips, and more in modern editors.

Follow the installation guide to get Coc.nvim up and running. Once Coc.nvim has been installed, you need to install some language server extensions to provide intellisense support for JavaScript and Typescript projects. You can so so with :CocInstall.

:CocInstall coc-tsserver coc-json

With coc-tsserver, you get so many features that make writing, debugging and refactoring JavaScript or Typescript a breeze. Without configuring anything, auto-completion should just work and you can see function signatures and relevant suggestions as you type.

You can even replace ALE’s linting and code formatting capabilities with the help of coc-eslint and coc-prettier, but as I’m already happy with my setup, I have opted not to do so at this time.

Wrap up

This post has covered several plugins that can help you get more utility from Vim when it comes to JavaScript development. While you don’t have to use any of this stuff to be productive with Vim, it can certainly help you avoid bugs, and speed up the development process.

I hope you’ve found this article useful. If you have any questions, or experience any troubles while configuring a plugin, do leave a comment below and I’ll get back to you. If you’re curious about my personal configuration, you can check out my dotfiles or send me a message on Twitter.

Thanks for reading!

Learn Node.js and MongoDB by building a URL Shortener app

Ayooluwa Isaiah — Mon, 21 Jan 2019 00:00:00 +0000

In this article, you’ll learn how to build a URL Shortener application with Node.js and MongoDB. Here’s a live demo of what we’ll be building. You can find the complete source code for this project in this GitHub repo.

Prerequisites

I assume basic familiarity with JavaScript as well as the command line. If you haven’t built a basic Node application before, you might want to start here first, then return to this tutorial at a later time.

You also need to have Node.js and npm installed on your computer. You can visit the Node.js website to view installation instructions for your operating system. npm comes bundled with Node, so once you install Node, you’ll have access to the npm command too.

The versions I used while building this project are as follows:

Node.js v11.2.0
npm v6.6.0

You can view the version of Node and npm you have installed by running the following commands in your terminal:

node -v
npm -v

Grab the starter files

Grab the starter files for this project at this GitHub repository. Clone the repo to your computer and cd into the created directory. Then run npm install from the project root to install all the dependencies specified in the package.json file. I’ll get into what each of the dependencies do later on as we progress.

Install MongoDB

MongoDB is a free and open-source NoSQL document database commonly used in modern web applications. You’ll need to have it installed on your machine. At the time of writing, the latest stable version is 4.0.5. This is the version I used throughout this tutorial.

Here are the installation instructions for Linux, macOS and Windows. If you’re on Ubuntu like myself, you can install MongoDB using apt:

sudo apt update
sudo apt install -y mongodb

You can checkout what version of mongoDB you have installed using mongo --version.

The database server should be automatically started after the installation process, but you should verify this before moving on from this step. On Ubuntu, you can use the following command to check the status of the mongoDB server:

sudo systemctl status mongodb

You should see this output:

If not, you can start it using the following command:

sudo systemctl start mongodb

Set up a basic Node server

Looking through the src folder within the project directory, you’ll see that we’ve got a server.js file and a public folder containing the markup and styles for the application frontend. The server.js file is where the bulk of the application code will be written in.

Unlike the previous Node tutorial where I used the built-in http module to set up the Node server, we’ll be using Express, a popular Node.js web application framework in this instance.

There are other web frameworks out there, but Express is simple enough, well documented and well supported so you shouldn’t run into many issues when using it in your applications.

If you look through the package.json file, you will see the express package is part of the dependencies that we installed earlier. Let’s go ahead and use it to set up the Node server in server.js:

const express = require('express');

const app = express();

app.set('port', process.env.PORT || 4100);
const server = app.listen(app.get('port'), () => {
  console.log(`Express running → PORT ${server.address().port}`);
});

You can start the server by running npm start in the terminal. I am making use of the Nodemon package to auto restart the Node server when changes are made to it so we don’t have to do so ourselves.

Set up the application frontend

As mentioned earlier, the app frontend lives in the public folder. We need to set up a new route on the server so that when a user visits the application, the HTML file will be sent and rendered in the browser.

Change the server.js file to look like this:

const express = require('express');
const path = require('path');

const app = express();

app.get('/', (req, res) => {
  const htmlPath = path.join(__dirname, 'public', 'index.html');
  res.sendFile(htmlPath);
});

app.set('port', process.env.PORT || 4100);
const server = app.listen(app.get('port'), () => {
  console.log(`Express running → PORT ${server.address().port}`);
});

path is a built-in module in Node.js. It allows us to link to directories and file paths in Node.js. The sendFile() method takes an absolute path to the file, so __dirname is used to avoid hardcoding the path. __ dirname is the directory in which the executing file is located, so path.join(__dirname, 'public', 'index.html') will resolve to src/public/index.html.

Navigate to http://localhost:4100 in your browser. Notice that the HTML is rendered correctly. However, the styles are missing even though style.css was linked correctly in index.html.

When the browser encounters the reference to style.css, it fires off a request to the server for that file. But since we haven’t configured our server to handle requests for static files (such as images, CSS and JavaScript), the server does nothing, and the request fails leaving the page without styles.

To fix this situation, we need to configure express to handle requests for static files correctly. We can do this using the a built-in middleware function in express as follows:

// beginning of the file

const app = express();

app.use(express.static(path.join(__dirname, 'public')))

// rest of the file

Now reload the page. It should work correctly:

Submit the form to the server

We need to write client side JavaScript to submit the contents of the form input to the server when a user submits the form.

We can do this without using any client side JavaScript by setting the action attribute of the form to a route on the server and setting the method attribute to POST, but I’ve opted to use JavaScript here so that we can handle the response and display the shortened url to the user without a full page refresh.

Create a new main.js file in the public directory and add the following code into it:

const form = document.querySelector('.url-form');
const result = document.querySelector('.result-section');
form.addEventListener('submit', event => {
  event.preventDefault();

  const input = document.querySelector('.url-input');
  fetch('/new', {
    method: 'POST',
    headers: {
      'Accept': 'application/json',
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      url: input.value,
    })
  })
    .then(response => {
      if (!response.ok) {
        throw Error(response.statusText);
      }
      return response.json();
    })
    .then(data => {
      while (result.hasChildNodes()) {
        result.removeChild(result.lastChild);
      }

      result.insertAdjacentHTML('afterbegin', `
        <div class="result">
          <a target="_blank" class="short-url" rel="noopener" href="/${data.short_id}">
            ${location.origin}/${data.short_id}
          </a>
        </div>
      `)
    })
    .catch(console.error)
});

This code listens for the submit event on the form, prevents the form submission and fires of a POST request to the server with the value of the form input in request body. The reason we wrap the body object in JSON.stringify is so that we can consume it as JSON on the server.

Note that we’re posting the data to the /new route which hasn’t been created on the server yet. We’ll create it in the next section. Before that, make sure you reference main.js in your index.html file before the closing body tag:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <meta http-equiv="X-UA-Compatible" content="ie=edge">
  <title>URL Shortener</title>
  <link rel="stylesheet" href="style.css" type="text/css">
</head>
<body>
  // rest of the code

  <script src="main.js"></script>
</body>
</html>

Access the form body on the server

Let’s go ahead and create the /new route that will process the URLs to be shortened. Add this below the root route in server.js:

app.post('/new', (req, res) => {

});

The first thing we need to do is access the JSON data that was sent from the client in the request body. To do this, we need to use of the body-parser package. This package parses all incoming request bodies and makes them accessible on req.body.

Since this package has been installed already, we can use it right away in server.js:

const express = require('express');
const path = require('path');
const bodyParser = require('body-parser');

const app = express();
app.use(express.static(path.join(__dirname, 'public')))
app.use(bodyParser.json());

// rest of the file

Now, we should be able to access the request body in the req.body property in the /new route:

app.post('/new', (req, res) => {
  console.log(req.body);
});

You can try this out by entering a URL into the form and submitting it. Then navigate to the terminal where your server is running to see the JSON data printed in the terminal.

This means we can access the url using req.body.url.

Validate the URL

Before we shorten the URL, we need to validate if the URL that was submitted is valid. Client side validation is handled for us in the browser because we’ve set the type of the input to url so the form won’t be submitted if the value doesn’t have a valid URL structure.

To make the application more robust, we need to validate the URL on the server as well. There are several npm packages that can handle this, but I’ve opted to do so using a few built-in Node modules.

The first thing we need to do is check if the URL has a valid structure, then we perform a DNS lookup to see if the domain is operational. A domain like https://google.com will pass both tests, but http://jidfsdm.com will fail the second one since that site does not exist.

Require the built-in dns module at the top of server.js:

const dns = require('dns');

Then change up the /new route as follows:

app.post('/new', (req, res) => {
  let originalUrl;
  try {
    originalUrl = new URL(req.body.url);
  } catch (err) {
    return res.status(400).send({ error: 'invalid URL' });
  }

  dns.lookup(originalUrl.hostname, (err) => {
    if (err) {
      return res.status(404).send({ error: 'Address not found' });
    };
  });
});

The URL class returns a new URL object with several properties if the input URL has a valid structure. Otherwise, it throws an error which we can catch and send back to the client.

If the URL input passes the first test, we then check to see if the domain is operational by passing the hostname portion of the URL (the domain) to dns.lookup which checks if the domain is live. If so, we can connect to our MongoDB instance and create the shortened version of the url as you’ll see.

Set up environmental variables

Environmental variables are a great way to configure how your program should work. They are key-value pairs that are stored on the local system where your program is being run, and are accessible from within your code.

It’s considered best practice to set app configuration data such as API keys, tokens, passwords and other sensitive details as environmental variables instead of hardcoding it into the program itself. This prevents you from accidentally exposing it to others, and also makes the values really easy to change without having to touch your code.

In Node.js, you can access variables defined in your environment via the process.env object. You can check the contents of this project via the Node.js REPL as shown below:

Aside from the operating system variables that are present by default, we can create project specific variables using a .env file. To avoid checking these files into source control, you should add .env to your .gitignore

Create a .env file at the root of your project directory and paste the following code into it:

DATABASE=mongodb://localhost:27017

Here, we’ve added the URL of our local MongoDB instance as an environmental variable. Port 27017 is the port that MongoDB runs on. The next thing to do is load the values defined in .env into process.env. The easiest way to do this is with the dotenv package which is already part of our app dependencies.

Add the following at the very top of server.js:

require('dotenv').config();

// rest of the file

This will read the contents of the .env file in the root of your project, parse its contents and initialize the values on process.env. Now, you’ll be able to access any set variable under process.env.<KEY>.

Connect to MongoDB

Let’s go ahead and connect to our local MongoDB instance in server.js as shown below:

require('dotenv').config()

const express = require('express');
const bodyParser = require('body-parser');
const path = require('path');
const dns = require('dns');
const { MongoClient } = require('mongodb');

const databaseUrl = process.env.DATABASE;

const app = express();
app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());

app.use(express.static(path.join(__dirname, 'public')))

MongoClient.connect(databaseUrl, { useNewUrlParser: true })
  .then(client => {
    app.locals.db = client.db('shortener');
  })
  .catch(() => console.error('Failed to connect to the database'));

// rest of the file

First, we’re importing MongoClient from mongodb which is the the native driver for interacting with a MongoDB instance in Node.js. Then we connect to the MongoDB instance specified in the DATABASE environmental variable.

If the connection is successful, we get a reference to the MongoDB instance client and can select a database using the client.db() method. Note that this method creates the database if it does not already exist.

Here, we are selecting a reference to the shortener database and storing that reference in app.locals which is an object provided by express. This object allows us to set local variables that persist throughout the life of the application, and can be accessed in other middleware functions (functions that have access to the req or res objects) via req.app.locals.

The reason we’re storing a reference to the database in app.locals.db is so that we can reuse the db object without having to open another connection to the MongoDB instance.

Shorten URLs

The next step is to actually shorten the URL and store it in the database. To create a unique short id for each url, we will be making use of the nanoid package.

Require it at the top of server.js below the other require statements:

const nanoid = require('nanoid');

Then create a new shortenURL function in server.js as follows:

// beginning of the file

MongoClient.connect(databaseUrl, { useNewUrlParser: true })
  .then(client => {
    app.locals.db = client.db('shortener');
  })
  .catch(() => console.error('Failed to connect to the database'));

const shortenURL = (db, url) => {
  const shortenedURLs = db.collection('shortenedURLs');
  return shortenedURLs.findOneAndUpdate({ original_url: url },
    {
      $setOnInsert: {
        original_url: url,
        short_id: nanoid(7),
      },
    },
    {
      returnOriginal: false,
      upsert: true,
    }
  );
};

// rest of the file

We need to reference a collection before we can add any data into the database. We can do so using the using the db.collection() method. If the collection does not exist yet, it’s created.

Before we shorten the URL and add it to the database, we need to check if the URL has not been shortened already to prevent duplicate database entries for one URL. We can do this using the findOneAndUpdate() method on the collection which allows us to modify a document that already exists in the database collection or create it if it doesn’t exist.

This method takes a few arguments. The first one is an object that is used to filter the collection. Here we are passing an object whose original_url property matches the url we are about to shorten. If a document in the database matches this filter, it will be returned and updated according to the update operators set in the second argument.

The $setOnInsert operator allows us to set the value of the document only if its being inserted. This means that the document will not be modified if it already exists, but if it doesn’t, it will be created with its value set to whatever we set in $setOnInsert.

In this case, the document will have two properties: original_url which is the url to be shortened, and short_id which is a unique 7 character id for that url.

We also need to set the upsert option to true. This ensures that the document is created if it doesn’t exist. Otherwise, $setOnInsert has no effect. Finally, setting returnOriginal to false ensures that findOneAndUpdate returns the new document if one is upserted, which is what we want in this case.

We can make use of the shortenURL() function in the /new route like this:

app.post('/new', (req, res) => {
  let originalUrl;
  try {
    originalUrl = new URL(req.body.url);
  } catch (err) {
    return res.status(400).send({error: 'invalid URL'});
  }

  dns.lookup(originalUrl.hostname, (err) => {
    if (err) {
      return res.status(404).send({error: 'Address not found'});
    };

    const { db } = req.app.locals;
    shortenURL(db, originalUrl.href)
      .then(result => {
        const doc = result.value;
        res.json({
          original_url: doc.original_url,
          short_id: doc.short_id,
        });
      })
      .catch(console.error);
  });
});

At this point, the document that was inserted will be sent to the client as JSON. It will be displayed on the page like this:

Set up a catch-all route for all shortened URLs

Apart from the / and /new routes, we need to handle the other requests for the shortened URLs. Specifically, we need to redirect them to the original URLs. Here’s how we can do so using Express:

app.get('/:short_id', (req, res) => {
  const shortId = req.params.short_id;

});

Here, we’re using named route parameters to capture the value of the short_id part of the URL. This value can then be accessed in the req.params object under the short_id property.

Once we have the short id, we need to check if a url with that short id exists in the database. Let’s create a new function for this purpose just below shortenURL:

const checkIfShortIdExists = (db, code) => db.collection('shortenedURLs')
  .findOne({ short_id: code });

The findOne method returns a document that matches the filter object passed to it or null if no documents match the filter.

We can then use the function in our catch-all route like this:

app.get('/:short_id', (req, res) => {
  const shortId = req.params.short_id;

  const { db } = req.app.locals;
  checkIfShortIdExists(db, shortId)
    .then(doc => {
      if (doc === null) return res.send('Uh oh. We could not find a link at that URL');

      res.redirect(doc.original_url)
    })
    .catch(console.error);
});

If the short id exists in our database collection, we will redirect the user to the original_url associated with that short id. Otherwise, an error message is sent to the user.

Now, you should be able to shorten long urls, visit the shortened url and be redirected to the original url.

Visualise your database with a GUI

A GUI will allow us to connect to our MongoDB instance and visualise all the data that is present. You also be able to create new data, update existing data and perform other similar operations.

There are several MongoDB GUIs out there, but the one I like to use is NoSQLBooster. It is available for Linux, macOS and Windows, and you can download it here.

Once you install and open the app, it should connect to your local MongoDB instance by default. Otherwise, you can click the Connect button in the top left and connect to it from there.

Once connected, you’ll able to view all the collections present in the database, and interact with the data present in those collections.

Deploy to Heroku

Before we deploy the app to Heroku, we need to create a cloud hosted MongoDB instance first. We can do so using MongoDB Atlas. Create an account at that link and create a new cluster when you are redirected to the Create New Cluster page.

Next, click the Security tab and hit the Add new user button. Give the user a name and password, then click the Add user button.

Following that, click IP Whitelist and then Add IP Address. This allows you to choose what IP addresses can access your cluster. Since this is just a demo application, you can allow access from anywhere by clicking the Allow Access From Anywhere button.

Next, go back to the Overview, and hit the Connect button. Choose Connect your Application and then Short SRV connection string. Take note of that string as we’ll be using it shortly on Heroku.

Head over to the Heroku website and sign up for a free account. Once your account is activated, follow this link to create a new app. Give it a unique name. I called mine freshman-shortener.

Next, follow the instructions here to install the Heroku CLI on your machine. Then run the heroku login command in the terminal to login to your Heroku account.

Make sure you’ve initialised a git repository for your project. If not, run the git init command at the root of your project directory, then run the command below to set heroku as a remote for your git repo. Replace <app name> with the name of your application.

heroku git:remote -a <app name>

Next, create a Procfile in the root of your project directory (touch Procfile) and paste in the following contents:

web: node src/server.js

Following that, specify the version of Node you are running in your package.json file under the engines key. I specified version 11.2.0 since that’s the version I’m running on my computer. You should change that value to match the version of Node you have on your machine.

{
  "name": "url-shortener",
  "version": "1.0.0",
  "description": "URL Shortener Demo",
  "main": "src/server.js",
  "scripts": {
    "start": "npx nodemon src/server.js"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/freshman-tech/url-shortener.git"
  },
  "keywords": [
    "url shortener",
    "mongodb",
    "nodejs"
  ],
  "author": "Ayo Isaiah",
  "license": "MIT",
  "bugs": {
    "url": "https://github.com/freshman-tech/url-shortener/issues"
  },
  "homepage": "https://github.com/freshman-tech/url-shortener#readme",
  "devDependencies": {
    "nodemon": "^1.18.9"
  },
  "dependencies": {
    "body-parser": "^1.18.3",
    "dotenv": "^6.2.0",
    "express": "^4.16.4",
    "mongodb": "^3.1.10",
    "nanoid": "^2.0.1"
  },
  "engines": {
    "node": "11.2.0"
  }
}

Before you deploy the app, head over to the Settings tab in the Heroku dashboard and hit Reveal Config Vars. This is where you will set the environmental variables for your app.

As we did earlier on the local .env file, enter DATABASE as the KEY and the SRV string from MongoDB Atlas as the value, then click Add. Remember to replace <PASSWORD> in that string with password for the user you created earlier.

Notice how easy it is to change the database location depending on the environment the program is running without touching the application code. This is one huge advantage of using environmental variables for project configuration.

Finally, commit your code and push it to the Heroku remote using the following commands:

git add .
git commit -m "Initial commit"
git push heroku master

Once the deployment process is done, you can open https://<your-app-name>.heroku.com to view and test your project.

Wrap up

We’ve successfully created a fully featured URL shortener and learnt the basics of Express and MongoDB along the way. We also learnt how to set up a cloud based MongoDB instance on MongoDB Atlas and how to deploy the app to Heroku.

I hope this exercise was helpful to you. If you have any questions regarding this tutorial, please leave a comment below and I’ll get back to you.

Thanks for reading!

Originally published at freshman.tech on January 24, 2019. Subscribe to my newsletter to get my latest tutorials on JavaScript and CSS delivered to your inbox.