DEV Community: James Edwards

Convert Markdown to HTML with Node.js

James Edwards — Thu, 05 Aug 2021 02:44:57 +0000

Unlike the name implies, Markdown is a markup language that can be used to create rich text output while authoring content in a plain text editor without formatting. Like HTML, Markdown includes a base syntax, however there is no formal specification for Markdown, like there is for HTML. As a result there are many Markdown variants, with each providing their own syntax variations and specifications.

While there are some differences among the flavors of Markdown, one rather nice aspect of authoring content with Markdown is that it can be readily converted to HTML using one of the many markdown processing technologies that are available. One way that can facilitate the creation of a website's HTML files, while still authoring content in Markdown, is to use Node.js to convert Markdown content into an HTML file. The resulting HTML output can then be uploaded to Jamstack website hosting, using static HTML files.

In this post we'll use Node.js and CLI commands to read a Markdown file, convert that file to an HTML string and then write the HTML string to a new file. When we have the file created, we can start a local development server to test the file in a web browser. Before following the steps make sure to have Node.js and npm installed.

Setup Node.js CLI Project

To start, setup the package.json file used with Node.js by running the command npm init in a terminal window open to your project folder. Then follow the prompts shown by the npm init process and a package.json file should have been created. With the package.json file in place we can run additional commands to install the npm packages that are used to convert Markdown to HTML.

npm install

In the same terminal window run the command npm install markdown-it highlight.js fs-extra cross-env rimraf @babel /cli @babel /core @babel /preset-env @babel /preset-typescript --save, followed by the command npm install typescript @types/node @types/markdown-it @types/fs-extra --save-dev.

After running both of these commands a new folder named "node_modules" should be present in your project folder. In the "node_modules" folder the following npm packages are installed:

Add Support For ES Modules

For this example these packages are also installed, mostly to support using TypeScript and ES modules in Node.js, which is optional.

The remainder of these steps will include setting up the TypeScript and Babel compilers to use ES Modules in Node.js for the CLI script that will convert Markdown into HTML, and write the HTML string to a file.

To support ES modules there is one more configuration that must be included in the package.json file. This is the "type" property with the value set to "module" as indicated below.

{
  "type": "module"
}

package.json Scripts

Additionally, we need to configure the "scripts" section of the package.json file to include the npm CLI scripts that will be used in the following steps. Since we are modifying the package.json file at this time go ahead and also add the following to the scripts property:

{
  "scripts": {
    "typecheck": "tsc --p .",
    "clean": "rimraf dist",
    "compile": "cross-env-shell babel src -d dist --source-maps --extensions '.ts'",
    "start": "npm run clean && npm run compile && node ./dist/index.js",
    "start-typecheck": "npm run typecheck && npm run start"
  }
}

These scripts are responsible for invoking the TypeScript and Babel compilers, to carry out typechecking and the compilation of TypeScript into JavaScript. These use most of the optional packages that were installed for that process. In a later step we can run these package.json scripts as CLI commands to first compile TypeScript and then run the JavaScript output with Node.js to convert Markdown into HTML.

package.json

With all the required packages installed and ES modules configured, the package.json file in your project should look like this:

{
  "name": "convertmarkdowntohtml",
  "type": "module",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "typecheck": "tsc --p .",
    "clean": "rimraf dist",
    "compile": "cross-env-shell babel src -d dist --source-maps --extensions '.ts'",
    "start": "npm run clean && npm run compile && node ./dist/index.js",
    "start-typecheck": "npm run typecheck && npm run start"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@babel/cli": "^7.14.8",
    "@babel/core": "^7.14.8",
    "@babel/preset-env": "^7.14.9",
    "@babel/preset-typescript": "^7.14.5",
    "cross-env": "^7.0.3",
    "fs-extra": "^10.0.0",
    "highlight.js": "^11.2.0",
    "markdown-it": "^12.2.0",
    "rimraf": "^3.0.2"
  },
  "devDependencies": {
    "@types/fs-extra": "^9.0.12",
    "@types/markdown-it": "^12.0.3",
    "@types/node": "^16.4.10",
    "typescript": "^4.3.5"
  }
}

If you are having trouble with the package install try copying the package.json from above and save that as your package.json file, then run the command npm install to install all of the listed packages.

Configure TypeScript Compiler with tsconfig.json

TypeScript is not required to convert Markdown to HTML, but it is not that much extra configuration to add when compared to the benefits of using TypeScript. Since the npm package for TypeScript was just installed we can add a new file to the project folder named "tsconfig.json" and this will contain the TypeScript compiler configuration settings that are recommended when using TypeScript and Babel in the same project.

{
  "compilerOptions": {
    "allowSyntheticDefaultImports": true,
    "isolatedModules": true,
    "strict": true,
    "module": "esnext",
    "lib": ["ES2019"],
    "noEmit": true,
    "moduleResolution": "node",
    "skipLibCheck": true
  },
  "include": ["src/**/*.ts"],
  "exclude": ["node_modules/**/*", "dist/**/*"]
}

The configuration will use TypeScript for type checking only, and the actual compilation of TypeScript into JavaScript will instead be carried out by the Babel compiler.

Configure Babel Compiler with babel.config.json

Just as the tsconfig.json file was added, we can add another file for the Babel configuration settings. This file is named "babel.config.json" and contains the following:

{
  "presets": [
    [
      "@babel/preset-env",
      { "modules": false, "targets": { "node": "current" } }
    ],
    ["@babel/preset-typescript"]
  ],
  "ignore": ["node_modules"]
}

The Babel compiler does not type check TypeScript code and will attempt to output valid JavaScript regardless of the TypeScript source. This is why the TypeScript compiler is used for type checking, and the benefit of using both is that the Babel compiler has presets available to ensure that the JavaScript generated will target a specific environment, in this case the current version of Node.js, and the "modules" property is set to false, which will preserve ES modules.

Create Markdown File

With our Node.js CLI project setup and package.json scripts already configured, the next part of the process to convert Markdown into HTML will be to create a sample Markdown file with a variety of content that includes the basic syntax shared among most Markdown flavors. To do this create a new folder for your project, named "content" and then inside the "content" folder create a new file named "index.md". When you have the index.md file created you can copy the sample Markdown content below into it.

# H1

## H2

### H3

#### H4

**bold text**

_italicized text_

> blockquote

1. First item
2. Second item
3. Third item

- First item
- Second item
- Third item

`code`

---

```javascript
function() {
  console.log("This is some javascript included in a markdown 
  code block, and it will be converted to valid HTML with code 
  syntax highlighting.");
}
```

<kbd>this is a keyboard input html element</kbd>

```html
<span>this will remain html even after the Markdown is 
converted to HTML</span>
```

[Dev Extent](https://www.devextent.com)

![Dev Extent](https://www.devextent.com/images/devextent.png)

Create Node.js CLI Script

Now that there is a Markdown file in the project we can add a new folder named "src" and in that folder add a new file named "index.ts". This is the Node.js script responsible for converting the Markdown file into an HTML file, and to start it looks like this:

(async function convertMarkdownToHtml() {
  console.log("Converting Markdown to HTML...");
})();

You can now run the command npm run start-typecheck or npm run start to compile without typechecking and you should see the console log is displayed. This means that the Node.js CLI project is working correctly, first compiling the TypeScript source code and then executing the generated JavaScript output with Node.js, all in one command.

Read Markdown File

After verifying that the Node.js CLI script is working correctly go ahead and add this code:

import fs from "fs-extra";

(async function convertMarkdownToHtml() {
  console.log("Converting Markdown to HTML...");

  // markdown source
  const content = await fs.readFile("./content/index.md", "utf8");
})();

The additional code imports one node module, the fs-extra package, and it provides the "readFile" function to asynchronously read the "index.md" file in the content folder. The contents of the Markdown file are then assigned to the variable named "content". We now have a string of Markdown content that is ready to be converted into HTML, and to do that the markdown-it package will be used.

Configure markdown-it Markdown Parser Options

To configure the markdown parser included in the markdown-it package, create a new folder in the "src" folder named "utils" and then in the "utils" folder create a new TypeScript file named "markdown.ts". In the "markdown.ts" the markdown-it package will be imported and the markdown parser object will be constructed and exported.

import MarkdownIt from "markdown-it";

const markdown: MarkdownIt = MarkdownIt({
  html: true,
});

export { markdown };

There is one configuration option passed into the markdown parser configuration and that is to support HTML tags in the markdown source. This is optional, and not required but it can be helpful to support using HTML for elements that are lacking in Markdown syntax.

Add Code Syntax Highlighting With highlight.js

Besides optionally supporting HTML tags in the Markdown source, the markdown parser included with the markdown-it package can apply syntax highlighting to designated code blocks. Make the following adjustments to the markdown.ts file to include this option:

import hljs from "highlight.js";
import MarkdownIt from "markdown-it";

const markdown: MarkdownIt = MarkdownIt({
  html: true,
  highlight: function (str, lang) {
    if (lang && hljs.getLanguage(lang)) {
      try {
        return (
          '<pre><code class="hljs">' +
          hljs.highlight(str, { language: lang, ignoreIllegals: true }).value +
          "</code></pre>"
        );
      } catch (__) {}
    }
    return (
      '<pre><code class="hljs">' +
      markdown.utils.escapeHtml(str) +
      "</code></pre>"
    );
  },
});

export { markdown };

The highlight.js module is able to dynamically determine the language syntax highlighting based on the "lang" variable valuable that is passed into the "highlight" function the highlight.js module API provides.

Instead of an error when encountering inconsistent syntax, the "ignoreIllegals" parameter configures the highlight.js highlighter to finish highlighting. You may wish to leave this option out, but there is discussion whether the default value of the "ignoreIllegals" options should be changed to true, as is used in this example.

If highlight.js cannot determined the language of the code block it will apply the "escapeHtml" function provided to the markdown string, and also wraps the code block section into a code element nested inside a pre element.

These additions will import the highlight.js module and apply the formatting required to dynamically highlight code blocks based on the language provided. The sample markdown file created in a previous step includes a block of JavaScript code that will have dynamic syntax highlighting applied when converted to HTML.

Convert Markdown to HTML with markdown-it parser

The "markdown.ts" file can now be imported in the "index.ts" file to access the Markdown parser with the previous configuration applied. To import the "markdown.ts" file and use the "render" function provided by the markdown-it module API, make these changes to the "index.ts" file:

import fs from "fs-extra";
import { markdown } from "./utils/markdown.js";

(async function () {
  console.log("Converting Markdown to HTML...");

  // markdown source
  const content = await fs.readFile("./content/index.md", "utf8");

  // converted to HTML
  const rendered = await markdown.render(content);
})();

The Markdown content, converted to HTML, is now assigned to the variable named "rendered". To view the rendered HTML you can output the "rendered" variable to the console and then run the command npm run start-typecheck, once more.

The contents of the "rendered" variable are valid HTML, but they do not represent an entire HTML document. To ensure that the Markdown source is converted into a complete and valid HTML document another variable is added, named "htmlFile", and this wraps the "rendered" variable string value in additional HTML code to create an entire HTML document. The "index.ts" should now look like this:

import fs from "fs-extra";
import { markdown } from "./utils/markdown.js";

(async function () {
  console.log("Converting Markdown to HTML...");

  // markdown source
  const content = await fs.readFile("./content/index.md", "utf8");

  // converted to HTML
  const rendered = await markdown.render(content);

  const htmlFile = `<!DOCTYPE html>
  <html lang="en">
  <head>
  <meta charset="UTF-8" />
  <title>Convert Markdown to HTML with Node.js</title>
  <link rel="stylesheet" href="./default.css">
  </head>
  <body>
  ${rendered}
  </body>
  </html>`;
})();

Note: The "default.css" file referenced in the head of the HTML document will be copied in the following step from the default style sheet theme included with the highlight.js npm package.

Write HTML File

Instead of writing this file in the project folder root, the fs-extra module includes a "mkdirs" function that can programmatically create a folder. Using this function a new folder will be created named "public", and the generated HTML file saved there.

The highlight.js module provides many different style sheet themes to choose from when applying code block syntax highlighting. For this example the "default.css" theme is used, and that file is copied from the highlight.js module, inside the "node_modules" folder into the public folder that is programmatically created for the generated HTML. This way when the style sheet is reference in the "index.html" file, the "default.css" file is available in the same folder.

import fs from "fs-extra";
import { markdown } from "./utils/markdown.js";

(async function () {
  console.log("Converting Markdown to HTML...");

  // markdown source
  const content = await fs.readFile("./content/index.md", "utf8");

  // converted to HTML
  const rendered = await markdown.render(content);

  const htmlFile = `<!DOCTYPE html>
  <html lang="en">
  <head>
  <meta charset="UTF-8" />
  <title>Convert Markdown to HTML with Node.js</title>
  <link rel="stylesheet" href="./default.css">
  </head>
  <body>
  ${rendered}
  </body>
  </html>`;

  await fs.mkdirs("./public");

  await fs.writeFile("./public/index.html", htmlFile, "utf8");

  await fs.copy(
    "./node_modules/highlight.js/styles/default.css",
    "./public/default.css",
    { overwrite: true }
  );

  console.log("HTML generated.");
})();

Run the command npm run start-typecheck once more and a new file "index.html" should be generated inside a new folder named "public" in your project folder, along with the "default.css" file that was copied from the "node_modules" folder.

You can now view the "index.html" file that will contain the Markdown source converted into HTML. The "index.html" file should look similar to this:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <title>Convert Markdown to HTML with Node.js</title>
    <link rel="stylesheet" href="./default.css" />
  </head>
  <body>
    <h1>H1</h1>
    <h2>H2</h2>
    <h3>H3</h3>
    <h4>H4</h4>
    <p><strong>bold text</strong></p>
    <p><em>italicized text</em></p>
    <blockquote>
      <p>blockquote</p>
    </blockquote>
    <ol>
      <li>First item</li>
      <li>Second item</li>
      <li>Third item</li>
    </ol>
    <ul>
      <li>First item</li>
      <li>Second item</li>
      <li>Third item</li>
    </ul>
    <p><code>code</code></p>
    <hr />
    <pre><code class="hljs"><span class="hljs-keyword">function</span>(<span class="hljs-params"></span>) {
  <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(<span class="hljs-string">&quot;This is some javascript included in a markdown code block, and it will be converted to valid HTML with code syntax highlighting.&quot;</span>);
}
</code></pre>
    <p><kbd>this is a keyboard input html element</kbd></p>
    <pre><code class="hljs"><span class="hljs-tag">&lt;<span class="hljs-name">span</span>&gt;</span>this will remain html even after the Markdown is converted to HTML<span class="hljs-tag">&lt;/<span class="hljs-name">span</span>&gt;</span>
</code></pre>
    <p><a href="https://www.devextent.com">Dev Extent</a></p>
    <p>
      <img
        src="https://www.devextent.com/images/devextent.png"
        alt="Dev Extent"
      />
    </p>
  </body>
</html>

You can validate the generated HTML code with the W3C Markup Validation Service, and you can also use the http-server npm package to create a local web server on your computer to view the "index.html" file in a browser.

View HTML File Locally

To test the Markdown converted into HTML, in a browser you can run the command npm install http-server --save-dev to install the http-server npm package. Then add the following to the package.json scripts property:

{
  "scripts": {
    "serve": "http-server"
  }
}

Then you can run the command npm run serve and the generated "index.html" file will be served from the public folder in your project. You should be able to navigate to "localhost:8080" and there you will see the content of the "index.html" with the styles from "default.css" applied to the syntax highlighted code block.

Send a POST Request Containing a GraphQL Query with the Fetch API

James Edwards — Sun, 06 Jun 2021 21:06:34 +0000

GraphQL is a query language specification that is used for Web APIs to permit the use of API clients to create data queries. The queries can be specific to the client, and they are sent to a GraphQL server that is able to return exactly the data that was requested. A single GraphQL POST request can be used to obtain all of the data that is needed for the current context. This is in contrast to RESTful APIs, that might result in a chain or waterfall of requests, with each request requiring data from the previous, in order to retrieve all of the data from the API server.

Typically a GraphQL client is used to facilitate the client side query building, and to send HTTP POST requests containing GraphQL queries to the GraphQL server responsible for returning the data. It is not required to use a dedicated GraphQL client, as it is possible to send a GraphQL query as a POST request using the Fetch API, and this is similar to the process used to submit FormData using the Fetch API. To show how to send a POST request containing a GraphQL query with the Fetch API, data from the GraphQL API: https://content.wpgraphql.com/graphql provided by WPGraphQL can be used. After fetching the latest posts from the GraphQL API, by sending a POST request containing the GraphQL query, we can display the data as a list with each item title as a link.

Create HTML File

First, create an HTML file that will link to a JavaScript file containing the code that will send the GraphQL query as a POST request with the Fetch API. After sending the POST request containing the GraphQL query, the result of the query will be displayed as HTML, and before any data is received a no data message is displayed. In the project folder add a new file named "index.html" with the following content:

<!-- index.html -->
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <title>Post a GraphQL Query with the Fetch API</title>
  </head>
  <body>
    <div id="data-container">
      <p>no data yet!</p>
      <button id="data-button">Get Data</button>
    </div>
    <script src="/script.js"></script>
  </body>
</html>

Add JavaScript File

In the "index.html" file there is a JavaScript file referenced that is named "script.js". We can create that file in the same folder as the index html file. After creating "script.js" in the project folder add the following code:

const dataFetchDisplay = function ({
  eventListenerSelector,
  eventType,
  dataFetcher,
  displayUpdater,
  dataTargetSelector,
}) {
  document
    .querySelector(eventListenerSelector)
    .addEventListener(eventType, async () => {
      displayUpdater(dataTargetSelector, await dataFetcher());
    });
};

The "dataFetchDisplay" function has an options object as the parameter that contains the information needed to send the Fetch API POST request containing a GraphQL query, although we have yet to call this function or define the functions "displayUpdater" and "dataFetcher" that are included in the options parameter and used within the async callback of the event listener that is instantiated, when the "dataFetchDisplay" function is called. Here is how the "dataFetchDisplay" function will be used:

dataFetchDisplay({
  eventListenerSelector: "#data-button",
  eventType: "click",
  dataFetcher: getData,
  displayUpdater: updateDisplay,
  dataTargetSelector: "#data-container",
});

Notice that the "eventListenerSelector" and "dataTargetSelector" parameters correspond to the ID attributes that are present in the index.html file created in the first step. These values can be changed, but the values must match the HTML document ID attributes. Go ahead and add the invocation of the "dataFetchDisplay" function directly below the function definition previously added to script.js.

Fetch API POST Request with GraphQL Query

Now that we have the "dataFetchDisplay" function defined and being called, if we try to run this code it will result in an error because the helper functions to get the data and display it are not yet defined. Directly above the "dataFetchDisplay" function add the following code to define the "getData" function that is referenced in the "dataFetcher" options object parameter key value.

const getData = async function () {
  return (
    await (
      await fetch("https://content.wpgraphql.com/graphql", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          query: "{ posts { nodes { title, link } } }",
        }),
      })
    ).json()
  ).data.posts.nodes;
};

The getData function shown above is where the POST request, sent by the Fetch API, containing the GraphQL query is defined. For this example the GraphQL API is provided by WPGraphQL, and the query will retrieve the link and title information for the ten most recent blog posts. Since we know the format of the data that is returned from the GraphQL query POST request sent with the Fetch API, we can return only the "nodes" in the "getData" function. That way when the "getData" function is used the data is already formatted as an array of post objects.

Display GraphQL Query Data

Now that we have the "getData" function defined and the GraphQL query data is returned after sending a POST request using the Fetch API, we need to display the data once it is returned from the GraphQL API server. To do this the function that is passed in as the "displayUpdater" parameter in the options object will be used. Add this code above the "dataFetchDisplay" function in the "script.js" file:

const updateDisplay = function (selector, data) {
  const list = document.createElement("ul");

  data.forEach(function (item) {
    const listItemLink = document.createElement("a");
    listItemLink.textContent = item.title;
    listItemLink.setAttribute("href", item.link);

    const listItem = document.createElement("li");
    listItem.appendChild(listItemLink);

    list.appendChild(listItem);
  });

  document.querySelector(selector).replaceChildren(list);
};

The "updateDisplay" accepts two parameters: one to indicate the target element to insert the HTML that is generated and the second is the data array. In this example a link item is created for each data object using the title. The list of link elements is then used to replace the html of the target element.

By passing the "getData" and "displayUpdater" functions in as parameters to the "dataFetchDisplay" function, both the query and the way it should be displayed can be changed to suit the usage context. The "dataFetchDisplay" function is generic in that sense as the parameters determine what data to display, and how,based on the specific usage of the function.

Putting all of the code sections together should result in a script.js file that looks like this:

//script.js

const getData = async function () {
  return (
    await (
      await fetch("https://content.wpgraphql.com/graphql", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          query: "{ posts { nodes { title, link } } }",
        }),
      })
    ).json()
  ).data.posts.nodes;
};

const updateDisplay = function (selector, data) {
  const list = document.createElement("ul");

  data.forEach(function (item) {
    const listItemLink = document.createElement("a");
    listItemLink.textContent = item.title;
    listItemLink.setAttribute("href", item.link);

    const listItem = document.createElement("li");
    listItem.appendChild(listItemLink);

    list.appendChild(listItem);
  });

  document.querySelector(selector).replaceChildren(list);
};

const dataFetchDisplay = function ({
  eventListenerSelector,
  eventType,
  dataFetcher,
  displayUpdater,
  dataTargetSelector,
}) {
  document
    .querySelector(eventListenerSelector)
    .addEventListener(eventType, async () => {
      displayUpdater(dataTargetSelector, await dataFetcher());
    });
};

dataFetchDisplay({
  eventListenerSelector: "#data-button",
  eventType: "click",
  dataFetcher: getData,
  displayUpdater: updateDisplay,
  dataTargetSelector: "#data-container",
});

Test GraphQL Post Request Locally

At this point we have the "index.html" and the "script.js" file setup so we can make sure that it is working by testing it locally. To do this we will need to install the http-server npm package. Before proceeding make sure to have Node.js and npm installed as they are required.

npm init package.json

Once installed you can open the project folder in a terminal window and run the npm init command, and follow the prompts that are displayed. This will set up the package.json in the project folder.

npm install http-server

After configuring the package.json file run the command npm install http-server --save-dev. The http-server npm package is now installed as a development dependency.

add npm script

In the "scripts" object of the package.json file configuration add the following script:

{
  "scripts": {
    "dev": "http-server"
  }
}

The dev script can now be run and this will start the local development environment using the http-server npm package. You should now have a "node_modules" folder that was added to the project folder, and the package.json file should look like this:

{
  "name": "post-graphql-query-fetch-api",
  "version": "1.0.0",
  "description": "",
  "main": "script.js",
  "scripts": {
    "dev": "http-server",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "http-server": "^0.12.3"
  }
}

To start the local development environment with http-server run the command npm run dev and navigate to the url that is displayed in the console output. The development url will most likely be "http://localhost:8080", as this is the default setting for the local server configuration.

After running the npm run dev command and navigating "http://localhost:8080" you should see the "no data yet" message in your browser and the "get data" button we created earlier. To send the GraphQL query POST request with the Fetch API click the "get data" button, and the latest ten posts should will display on the page.

In some cases it might be beneficial to include a dedicated GraphQL client in your project, but in others using the Fetch API to send a POST request containing a GraphQL query without a GraphQL client can be sufficient. This can save time if the other more advanced features that come with GraphQL clients are not needed, especially if requests to the GraphQL server are infrequent.

Deploy a Jamstack website to Azure Blob Storage with GitHub Actions

James Edwards — Sat, 17 Apr 2021 23:30:54 +0000

GitHub Actions are included with Github Repositories and can be used to automate project workflows like building and deploying code. In this example we will see how to automate the build process and deployment of a site built with the Jamstack. We can use GitHub Actions to checkout a specific branch in a git repository, and then execute a build process that is common to Jamstack sites that are created with a static site generator like Next.js or Eleventy. On completion of the static site generator build process, the static site folder will then be uploaded to Azure Blob Storage, utilizing the static website hosting feature included with Azure Blob Storage.

Azure Blob Storage Static Website Hosting

In order to use the static website hosting feature included with Azure Blob Storage you need to create an Azure account if you do not already have one. Once you have created your account and logged in you will need to create the Storage Account resource that will provide the Blob Storage service. In the Azure portal select create a new resource and then search for "storage account". Then click create and follow the setup steps that are displayed to give the storage account a name and region. You can leave any pre-configured settings as the default setting.

Within the Azure portal navigate to your newly created storage account and in the left side navigation find the section labelled "Settings" and then within that section select the "Static website" feature.

Configure Index Document Name and Error Document Path

In the static website settings, enabling the static website feature will automatically generate the primary endpoint based on the storage account name. In the "Index document name" and "Error document path" fields enter the following:

Based on how your static website is configured, or the static site generator that you are using, the "Error document path" might be different than what is shown for this example, however the "Index document name" will most likely remain as "index.html". Make sure to update these settings to correspond to the configuration that you are using. The storage account static website is now available and you can go to the primary endpoint and you will see this:

This is good and it means that the static website is setup and publicly available. Now we can setup GitHub Actions to automated deployments anytime there is a commit pushed to the main branch in our GitHub repository.

Create Remote GitHub Repository and Configure Local Repository Remote

If you haven't already go ahead and create a new repository for your project, and after doing so follow the directions that are displayed to create a new repository from the command line on your local computer. This should result in a new repository that can push to the remote repository created on GitHub and any commits that we make will be applied to the main branch which is named "main". Make sure that you are able to push commits to GitHub and you can see them in your GitHub remote repository interface before proceeding.

Note: The following steps that configure the GitHub Actions workflow require that the branch name is set to "main".

Add .gitignore File

Additionally you will want to add a ".gitignore" file to your project that minimally contains:

node_modules
_output

The "_output" folder is included because this is created by the static site generator build process that is configured in the next step. This folder will always contain generated files so it does not need to be tracked by git.

Setup Static Site Generator

If you already have a repository, that includes a Jamstack site setup with a static site generator, you can proceed without this step. In order to illustrate the git-centric deployment process that is promoted by GitHub Actions we can use, morphic a static site generator built with Node.js and TypeScript. You don't need to use a static site generator in order to use GitHub Actions or the Static Website feature included with Azure Blob Storage, however it can be useful since GitHub Actions can execute commands that will invoke the static site generator build process and upon completion carry out the cloud deployment.

Install Node.js and Configure package.json

Before proceeding make sure to have Node.js and npm installed.

In your project folder run the command npm init and follow the prompts that are displayed to create a package.json file for your project. Then run the command npm install @protolith/morphic. This will install the morphic static site generator and its dependencies.

Add Static Website Content as Markdown Files

In the project folder where the package.json file was created you can then run this series of commands to create some site content.

mkdir content/pages

mkdir templates

cd content/pages

echo '---
title: "Home"
---
# <%%= model.title %%>
home page content

<a href="/about/">Go to About Page</a>' > index.md

echo '---
title: Page Not Found
---

The page you are looking for may have been moved or deleted.

[Go to Homepage](/)' > 404.md

echo '---
title: About
---
# <%%= model.title %%>
This is the about page.

<a href="/">Go to Home Page</a>' > about.md

cd ../../templates

echo '<!DOCTYPE html>
<html>
  <head>
    <title><%%= model.title %%></title>
  </head>
  <body>
    <%%- model.content %%>
  </body>
</html>' > index.ejs

cd ..

Note: make sure the files created (both ".md" and ".ejs" files) are encoded as UTF-8, or the static site generation process will not work. In Visual Studio Code you can change the file encoding by selecting the file and then in the bottom right hand corner toolbar the encoding is displayed. Click the encoding display in the bottom right toolbar to save with the UTF-8 encoding if needed.

Build and Serve Static Site Generator Locally

To make sure that morphic is setup correctly and generating the appropriate files you can run the command npx morphic --serve, and a browser window should open with the home page displaying a link to the about page. You can navigate back and forth from the home page and about pages. Also you can verify the 404 page was generated by going to the path "404.html" so the full url when running locally might be "http://localhost:3000/404.html", which corresponds to the error document path setting configured in the Azure Blob Storage Static Website settings.

Github Actions Workflow To Deploy Static Site to Azure Blob Storage

We now have our static site configured and ready to deploy with a GitHub Actions Workflow. To create a new GitHub Actions Workflow, that will build and deploy the static site from our GitHub repository, add a new folder within the current project folder named ".github". In the ".github" folder add another folder named "workflows" and within the "workflows" folder create a new workflow YAML file named "main.yml". The "main.yml" file is where we will create the GitHub Actions workflow that will build and deploy a Jamstack site to the Azure Blob Storage Static website. Here is the complete reference documentation for creating workflows provided by GitHub. We will only use a subset of the available features in our "main.yml" file to build and deploy on a git push command to the main branch in the repository we created. In the "main.yml" file add the following code:

name: MAIN

on:
  push:
    branches: [main]

env:
  BUILD_COMMAND: npx morphic

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
        with:
          fetch-depth: 0

      - name: Install Dependencies
        run: npm ci

      - name: Build
        run: ${{ env.BUILD_COMMAND }}

      - name: Update Blobs
        uses: azure/cli@v1.0.0
        with:
          inlineScript: |
            az storage blob sync -c '$web' -s _output --connection-string '${{ secrets.AZURE_STORAGE_CONNECTION_STRING }}'

This workflow will be activated anytime there is a git push command for the main branch in the remote git repository. It will setup a GitHub Actions environment that is running on the latest version of the Ubuntu operating system, and then checkout the latest version of the code from the main branch. After checking out the main branch, the workflow invokes the npm ci command, which is similar to the npm install command, and is suitable for continuous integration environments. This is required to install the morphic static site generator npm package, similar to how we ran the static site generator locally in a previous step.

With the package.json dependencies installed the "Build" step in the workflow will run the "BUILD_COMMAND" specified as an environment variable. In our case, with the morphic static site generator, this command is the same one as we used before npx morphic, except this time the --serve flag is omitted since the files will be deployed to Azure Blob Storage Static Website hosting.

After running the Build command the workflow will have access to the site output folder: "_output" in our case, and this is what will be used in the Azure CLI portion of the workflow with name "Update Blobs". This uses the Azure CLI Action to permit the use of Azure CLI commands. In this case we want to use the az storage blob sync command to ensure that "$web" blob container is synced to the the current build "_output" folder that was just created during the Build step of the workflow file.

In order to sync the files to the Azure Blob container named "$web", we need to pass in the "--connection-string" value that acts an access password for the storage account. Instead of creating a security risk by including the blob storage connection string directly in our publicly available workflow we can use GitHub Actions Repository Secrets to store secret variables that only the automated workflow environment can access.

Git Commit Project and Workflow Files

The workflow configuration file is now included in the project, so we can commit the example static files we created and the workflow file using the commands git add . and then git commit -m 'adding website files and workflow'. However, before we push these changes to the remote GitHub repository we need to configure the "AZURE_STORAGE_CONNECTION_STRING" GitHub Actions secret value that is referenced in the workflow file.

Github Actions Secret

In the Azure portal, the storage account connection string can be copied by going to the Storage Account that was created previously. In the left sidebar find the settings section that contains the Access keys configuration.

You can choose the connection string for key1 or key2, but make sure to use the connection string field value and not the Key field value.

Copy the connection string from the Azure portal settings and go to your GitHub repository interface, and in the top navigation there will be a section labelled "Settings". Clicking on "Settings" will bring you to a new page with a left side navigation that contains a section labelled "Secrets". This is where we will use the copied Azure Blob storage connection string to create a new GitHub Actions secret. On the Secrets page click the button to add a "New repository secret".

Then add the secret name that matches the variable name from the "main.yml" workflow file, in this example it is AZURE_STORAGE_CONNECTION_STRING, and paste the Azure Storage connection string into the Value field.

Then click "Add Secret" and our GitHub Actions Workflow can now access the Azure Storage connection string to sync files to Azure Blob Storage. Our GitHub Actions secret is now configured and ready for use.

Git Push to Deploy Static Website with GitHub Actions to Azure Blob Storage Static Website

Now you can run the command git push -u origin main from your local project and you should see the commit made previously has been pushed to the remote repository in the GitHub interface. In the top navigation of the GitHub repository, the "Actions" tab should now show the workflow automation is underway.

When the workflow build and deployment process completes, navigate to the primary endpoint for the Azure Blob Storage static website, that was displayed in the first step.

You should see the home page now displays the content of our site and you can navigate to the About page that was created earlier. If you accidentally navigate to a page that does not exist, the error document path is set to 404.html, and Azure Blob Storage will return the 404.md page content that was added as a page within the static site.

Anytime you want to make changes to your site in the future, you can follow the same process as this example by first committing the changes to the main branch and then pushing the changes to the remote GitHub repository. This will trigger the GitHub Actions workflow and the changes will be automatically updated on the static website.

Convert JSON to XML with the XML npm package

James Edwards — Wed, 31 Mar 2021 18:14:51 +0000

XML is a textual data format that is standardized, and as a result is widely used throughout a variety of systems. Two common usages are for website sitemaps and RSS feeds, both of which can use XML as the document format. Other usages of XML can include RESTful HTTP API endpoints, both receiving and returning XML requests and responses. This post will include the steps to convert JSON to XML with the XML npm package. First we will read a JSON file, convert the JSON object to an XML string, and then write the XML string to a file. Besides reading and writing files, the XML npm package can be used in other scenarios, where no files are involved as long as incoming data format is JSON and the desired data output format is an XML string.

npm init package.json

You will not need to to do this if you have an existing Node.js project setup but if you do not, then make sure to first install Node.js and npm. Then in a terminal window open new folder for the project and run the command npm init, and follow the prompts that are displayed. The package.json file should have been added to the project folder.

We also need to make one addition to the package.json file after it has been generated so that ES Modules can be used with Node.js. To support ES Modules add a "type" property to the package.json file object with the value set to "module". In the following steps we will configure the TypeScript compiler to output JavaScript using the ES Module format.

npm install

With the package.json generated we can run additional commands to install the npm packages we will use. In the same project folder run the command npm install xml typescript --save, this will install the XML and TypeScript npm packages. After that run another command npm install @types/xml --save-dev. This will install the TypeScript type definitions for the XML npm package. Your package.json should look similar to this:

{
  "type": "module",
  "name": "convertjsontoxml",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "typescript": "^4.2.3",
    "xml": "^1.0.1"
  },
  "devDependencies": {
    "@types/xml": "^1.0.5"
  }
}

Compile TypeScript

Now that we have installed the XML and TypeScript npm packages installed, we can configure TypeScript to compile our code to use with Node.js, by adding an npm package.json script. To do this add the following the the "scripts" property in the package.json file that was created in the first step:

{
  "scripts": {
    "compile": "tsc --allowSyntheticDefaultImports --isolatedModules --moduleResolution node --module esnext index.ts"
  }
}

The compile command will invoke the TypeScript compiler with the CLI flags that will generate the JavaScript output using the ES Module format. This will match the "type" property set to "module" in the package.json configured earlier. You can run this command using npm run compile in the terminal window.

Create Node.js Script

Next we can create a Node.js script, and as referenced in the package.json scripts "compile" command, the name of this file is index.ts. Here we will write the TypeScript code that will use the XML npm package to generate an XML string from a JSON object. In the index.ts file add the following code:

import { promisify } from "util";
import { readFile, writeFile } from "fs";
import xml from "xml";

const readFilePromise = promisify(readFile);
const writeFilePromise = promisify(writeFile);

(async function convertJsonToXml() {})();

This will set up the import statements for the XML npm package and also import the readFile and writeFile functions from the node fs module. Since these functions use callbacks by default, the promisify function is imported from the util module to convert the readFile and writeFile functions into promises. This way we can use async/await syntax.

Read JSON File

In the ConvertJsonToXml function the first thing we can do is read the JSON file containing a sample JSON object that we can convert to an XML string. Create a new file named "data.json" in the same project folder, and add this sample JSON object:

[
  {
    "name": "Next.js",
    "language": "JavaScript",
    "templates": "React",
    "description": "A framework for statically-exported React apps (supports server side rendering)"
  },
  {
    "name": "Gatsby",
    "language": "JavaScript",
    "templates": "React",
    "description": "Build blazing fast, modern apps and websites with React"
  },
  {
    "name": "Nuxt",
    "language": "JavaScript",
    "templates": "Vue",
    "description": "A minimalistic framework for serverless Vue.js applications."
  }
]

In the index.js file, inside of the ConvertJsonToXml function we can add this code to read the JSON file and parse it into a JSON object with the corresponding type signature:

const staticSiteGeneratorData = JSON.parse(
  await readFilePromise("data.json", "utf8")
) as [
  {
    name: string;
    language: string;
    templates: string;
    description: string;
  }
];

Once the json file is read and saved as the "staticSiteGeneratorData" variable we can use the Array.prototype.map() method to shape the data into the format we need, in order to use the XML npm package to convert the JSON object into an XML string. Below the code that is reading the data.json file add this code:

const xmlFormattedStaticSiteGeneratorData = [
  {
    staticSiteGenerators: [
      ...staticSiteGeneratorData.map((item) => {
        return {
          staticSiteGenerator: [
            {
              _attr: {
                language: item.language,
                templates: item.templates,
                description: item.description,
              },
            },
            item.name,
          ],
        };
      }),
    ],
  },
];

The result of the data that is assigned to the "xmlFormattedStaticSiteGeneratorData" variable will look like this:

[
  {
    "staticSiteGenerators": [
      {
        "staticSiteGenerator": [
          {
            "_attr": {
              "language": "JavaScript",
              "templates": "React",
              "description": "A framework for statically-exported React apps (supports server side rendering)"
            }
          },
          "Next.js"
        ]
      },
      {
        "staticSiteGenerator": [
          {
            "_attr": {
              "language": "JavaScript",
              "templates": "React",
              "description": "Build blazing fast, modern apps and websites with React"
            }
          },
          "Gatsby"
        ]
      },
      {
        "staticSiteGenerator": [
          {
            "_attr": {
              "language": "JavaScript",
              "templates": "Vue",
              "description": "A minimalistic framework for serverless Vue.js applications."
            }
          },
          "Nuxt"
        ]
      }
    ]
  }
]

Convert JSON File to an XML String

The JSON data assigned to the "xmlFormattedStaticSiteGeneratorData" variable, is now in the appropriate format to use with the XML npm package. Directly below the code that formats the data, and inside the "convertJsonToXml" function, add the following code:

const staticSiteGeneratorXmlString = xml(xmlFormattedStaticSiteGeneratorData);

The format of the xml string assigned to the "staticSiteGeneratorXmlString" is going to look like this:

<staticSiteGenerators>
    <staticSiteGenerator language="JavaScript" templates="React" description="A framework for statically-exported React apps (supports server side rendering)">Next.js</staticSiteGenerator>
    <staticSiteGenerator language="JavaScript" templates="React" description="Build blazing fast, modern apps and websites with React">Gatsby</staticSiteGenerator>
    <staticSiteGenerator language="JavaScript" templates="Vue" description="A minimalistic framework for serverless Vue.js applications.">Nuxt</staticSiteGenerator>
</staticSiteGenerators>

Write XML File

The XML string assigned to the variable "staticSiteGeneratorDataXmlString" can be written to an XML file with the writeFile module that we imported and promisified at the beginning of the index.ts file. To write the XML string to a file in the same project folder add this code below the XML npm package usage that was included in the prior step:

await writeFilePromise("data.xml", staticSiteGeneratorXmlString, "utf8");

Put all the code together and the index.ts file should look like this:

import { promisify } from "util";
import { readFile, writeFile } from "fs";
import xml from "xml";

const readFilePromise = promisify(readFile);
const writeFilePromise = promisify(writeFile);

(async function convertJsonToXml() {
  const staticSiteGeneratorData = JSON.parse(
    await readFilePromise("data.json", "utf8")
  ) as [
    {
      name: string;
      language: string;
      templates: string;
      description: string;
    }
  ];

  const xmlFormattedStaticSiteGeneratorData = [
    {
      staticSiteGenerators: [
        ...staticSiteGeneratorData.map((item) => {
          return {
            staticSiteGenerator: [
              {
                _attr: {
                  language: item.language,
                  templates: item.templates,
                  description: item.description,
                },
              },
              item.name,
            ],
          };
        }),
      ],
    },
  ];

  const staticSiteGeneratorXmlString = xml(xmlFormattedStaticSiteGeneratorData);

  await writeFilePromise("data.xml", staticSiteGeneratorXmlString, "utf8");
})();

Run Node.js Script with npm package.json Scripts

To test this code out and run the Node.js script we can add another package.json script that will first compile the TypeScript into JavaScript and then run the JavaScript output with Node.js. In the package.json file add a new package.json script named "start" that looks like this:

{ "scripts": { "start": "npm run compile && node index.js" } }

To use the start script run the command npm run start and you should then see the XML file generated and saved to the project folder. The contents of this file should match the format of the XML string shown previously. Anytime you want to change the data or formatting make sure to run the npm run start again to regenerate the data.xml file.

The XML npm package is a convenient way to convert JSON into XML, as long as the JSON data is formatted appropriately, or there is a step involved to properly format the original JSON data source into the format the XML npm package requires. For other usages of the XML npm packages you can read my other posts showing how to generate an XML sitemap and generate an XML RSS feed, like this example, both of these posts are using Node.js and npm.

Create a Service Worker with TypeScript

James Edwards — Tue, 23 Mar 2021 18:21:32 +0000

The Service Worker API is available to use in all major browsers. Service Workers are JavaScript files that contain event driven worker code, that do not run on the main thread of the browser. They are used as a proxy for network requests, more specifically, service workers can be used to intercept requests and modify them as well as cache responses. In addition to caching responses a service worker can be used to enhance the user experience of your website by displaying an offline page when there is no network connection available.

In this example, TypeScript is used to create a service worker with a network first then cache, caching strategy, to support viewing previously visited pages when offline. When there is no network connection available and a previous page version is not cached, the service worker will display an offline page.

Configure TypeScript and Babel with ES Modules

Before adding the TypeScript code for the service worker there are some prerequisite steps to configure the TypeScript compiler. If you want to include Typescript into your project you can view these posts:

to find more information about using TypeScript. The configuration will include Babel for TypeScript compilation as shown in the second post above with some minor adjustments, to support browser use rather than Node.js, made to the Babel configuration.

Configure package.json Scripts

Create a package.json file by running npm init and then run the command npm install typescript cross-env @babel/cli @babel/core @babel/preset-env @babel/preset-env @babel/preset-typescript --save. You then need to add three package.json scripts that look like this:

{
  "scripts": {
    "typecheck": "tsc --p .",
    "compile": "cross-env-shell babel $INIT_CWD -d $INIT_CWD --extensions '.ts' --no-comments --source-maps",
    "typecheck-compile": "npm run typecheck && npm run compile"
  }
}

Configure TypeScript for Type Checking Only

The package.json "typecheck" script will invoke the TypeScript compiler and only is responsible for type checking the TypeScript code, and it will not emit any JavaScript. The TypeScript compiler configuration options are specified in a file named "tsconfig.json". You can create this file with the following settings:

{
  "compilerOptions": {
    "allowSyntheticDefaultImports": true,
    "isolatedModules": true,
    "strict": true,
    "module": "esnext",
    "lib": ["es2019", "es6", "dom", "webworker"],
    "noEmit": true,
    "moduleResolution": "node",
    "skipLibCheck": true
  },
  "include": ["*.ts"],
  "exclude": ["node_modules/**/*"]
}

Notice that the "lib" property contains a value for "webworker" among the values. This is important because it indicates to the TypeScript compiler that it should load the type definitions for the Worker APIs.

The package.json "compile" script uses Babel to compile TypeScript and will be responsible for outputting JavaScript. The Babel compiler does not type check the TypeScript source code, so if there are any errors present the Babel compiler may attempt to provide output rather than indicating there was an error. Also the Babel compiler CLI command requires a source folder and output folder to be specified so this is set to the project folder with the "$INIT_CWD" npm CLI variable, for both the source and output folders. This way the JavaScript files will be written to the same folder as the TypeScript files.

Configure Babel with babel/preset-typescript and babel/preset-env

We are going to target browsers that are using ES Modules with the preset-env preset, in addition to using the preset-typescript preset. You can configure this by adding a babel.config.json file in the same folder as the package.json with the following settings:

{
  "presets": [
    [
      "@babel/preset-env",
      {
        "modules": false,
        "targets": { "esmodules": true }
      }
    ],
    ["@babel/preset-typescript"]
  ],
  "ignore": ["node_modules"],
  "comments": false,
  "minified": true
}

The presets are applied in the reverse order that they are listed in the Babel.config.json, so the preset-env changes will be applied to the JavaScript output created by preset-typescript.

TypeScript Type Checking with Babel Compilation

To combine both the typecheck and compile scripts you can run the package.json typecheck-compile command: npm run typecheck-compile. This will first use the TypeScript compiler to typecheck the code and then use the Babel compiler to generate JavaScript for the browser. If there are any type checking errors the command will fail and they will be shown in the console.

Register Service Worker

Now that we have TypeScript and Babel configured to use ES Modules, but before creating the actual service worker code, we need to write some code that will register the service worker in the browser. This code can be included in the HTML source directly, or in a script tag. If you are including the code with a script tag make sure to reference the JavaScript output file that is generated as a result of the TypeScript compilation process and not the TypeScript source file. We can name this file "script.ts" and place it in the root of the project.

export default class Main {
  constructor() {
    if ("serviceWorker" in navigator) {
      navigator.serviceWorker
        .register("/service-worker.js", { scope: "/" })
        .then(function () {
          console.log("Service Worker Registered");
        });
    }
  }
}

new Main();

HTML include script type="module"

You can include this script in an HTML file by creating a new file named "index.html", and save it in the root of the project, with this content:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <title>Service Worker Example</title>
  </head>
  <body>
    This page is loading a service worker!
    <script type="module" src="/script.js"></script>
  </body>
</html>

When the browser loads the HTML page the JavaScript above will attempt to load the service worker. If the browser does not support service workers the code will be skipped.

Besides the index.html file we are also going to need an HTML file for an offline page. In the same folder that the index.html file is saved create a new file name "offline.html" and add this content to show an offline state:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <title>You are not connected to the internet</title>
  </head>
  <body>
    You are not connected to the internet!
    <script type="module" src="/script.js"></script>
  </body>
</html>

When the service worker code is added in the following step the offline page will be automatically cached for future usage.

Set up Service Worker

Registering the service worker is what invokes the service worker code that is not running on the main thread. Inside of the service worker we will write code to listen for certain types of events that are triggered by the browser. These events are:

We can add the install event first by adding the following code to a file named "service-worker.ts". You can use a different name if you want, but make sure that it matches what is included in the "navigator.serviceWorker.register" function used in the previous step. In the service-worker.js file add the following code:

/// <reference lib="WebWorker" />

// export empty type because of tsc --isolatedModules flag
export type {};
declare const self: ServiceWorkerGlobalScope;

This is the beginning code for using a service worker with TypeScript and Babel compilation. The first line is a triple-slash directive to indicate to the TypeScript compiler to use the Worker API type definitions. This is why it was important to note the inclusion of the Worker API type definitions in the "lib" setting of the tsconfig.json file.

After loading the Worker API type definitions, the next line is and empty type export that prevents an error as a result of the typescript compiler option "isolatedModules". Using "isolatedModules" is recommend when using Babel with TypeScript, but this requires all modules to have either one export or import statement. Since the service worker is a standalone script it cannot import or export modules, and by including the empty type export, the typescript compiler will no longer show the error:

(error TS1208: All files must be modules when the '--isolatedModules' flag is provided)[]

The empty type export will be removed when they TypeScript code is compiled into JavaScript with Babel. This is a workaround until service workers have module support, and only needed when using the "--isolatedModules" flag with TypeScript.

The following declaration allows the type "ServiceWorkerGlobalScope" included in the WebWorker API type definitions, to be specified for the "self" variable that already exists. Without the type declaration the self variable would have the type "Window & typeof globalThis", and the type definitions of that type do not overlap with the "ServiceWorkerGlobalScope" type. This is because the service worker does not have access to the window object or any other global scope, and the self variable used in the service worker code is of type WorkerGlobalScope.self, rather than Window.self.

Similar to the empty type export shown above this is also a workaround that is only needed to inform the TypeScript compiler of the proper types that are being used. There is some discussion regarding the type signatures on the TypeScript Github repository:

Service Worker install Event

The first event listener that is added to the code, listens for the service worker install event to be triggered. When the install event occurs a cache using a combination of the "cacheName" and "version" variables is created, if it does not exist, and the index and offline HTML pages will be automatically added to the cache. Add this code below the "self" variable declaration:

const cacheName = "::yourserviceworker";
const version = "v0.0.1";

self.addEventListener("install", function (event) {
  event.waitUntil(
    caches.open(version + cacheName).then(function (cache) {
      return cache.addAll(["/", "/offline"]);
    })
  );
});

Service Worker activate Event

Below the install event listener code we can add a separate event listener that will be triggered on the activate event. The activate event occurs after the install event, and is used to clean up any old service worker caches.

self.addEventListener("activate", function (event) {
  event.waitUntil(
    caches.keys().then(function (keys) {
      // Remove caches whose name is no longer valid
      return Promise.all(
        keys
          .filter(function (key) {
            return key.indexOf(version) !== 0;
          })
          .map(function (key) {
            return caches.delete(key);
          })
      );
    })
  );
});

On activation if the "version" variable declared at the top of the service worker has been updated, any existing caches that do not match the new variable value will be deleted. This makes sure that the service worker is always using the latest cache values. The version variable acts as a cache bust when making changes to the service worker, after it has been previously deployed, otherwise the cached values will continue to be used until the service worker is no longer registered.

Service Worker fetch Event

After install and activation the service worker will listen for fetch events. In the fetch event listener function we can intercept responses and add them to the cache after the fetch response is complete, or the network can be bypassed and the response can be returned directly from the cache. The fetch event listener can be included below the activate event listener like this:

self.addEventListener("fetch", function (event) {
  const request = event.request;

  // Always fetch non-GET requests from the network
  if (request.method !== "GET") {
    event.respondWith(
      fetch(request).catch(function () {
        return caches.match("/offline");
      }) as Promise<Response>
    );
    return;
  }
});

If the request is not a GET request, for example it could be a POST or PUT request, the service worker will always defer these requests to the network and the response is never cached. In the event that there is no network connection and the network request fails the service worker will return the offline page for any requests that are not GET requests.

Network First then Cache - HTML Caching Strategy

Any HTML request will use a network first then cache, caching strategy. This enables the latest version of the page to be requested from the origin server if there is a network connection. Without a network connection, the service worker will check if a prior version of an HTML page is available in the cache, and if not the offline page will be returned. Below the non-GET requests conditional block add this code to proxy HTML requests:

// For HTML requests, try the network first, fall back to the cache,
// finally the offline page
if (
  request.headers.get("Accept")?.indexOf("text/html") !== -1 &&
  request.url.startsWith(this.origin)
) {
  // The request is text/html, so respond by caching the
  // item or showing the /offline offline
  event.respondWith(
    fetch(request)
      .then(function (response) {
        // Stash a copy of this page in the cache
        const copy = response.clone();
        caches.open(version + cacheName).then(function (cache) {
          cache.put(request, copy);
        });
        return response;
      })
      .catch(function () {
        return caches.match(request).then(function (response) {
          // return the cache response or the /offline page.
          return response || caches.match("/offline");
        });
      }) as Promise<Response>
  );
  return;
}

Cache First then Network - non-HTML Caching Strategy

For non-HTML requests, first the cache is checked and if the resource is in the cache it will be returned. If the non-HTML requests is not in the cache it will be requested from the origin server as long as there is a network connection, and then the response will be added to the cache for future requests. Add this code below the code that handles HTML requests:

// For non-HTML requests, look in the cache first, fall back to the network
if (
  request.headers.get("Accept")?.indexOf("text/plain") === -1 &&
  request.url.startsWith(this.origin)
) {
  event.respondWith(
    caches.match(request).then(function (response) {
      return (
        response ||
        fetch(request)
          .then(function (response) {
            const copy = response.clone();

            if (
              copy.headers.get("Content-Type")?.indexOf("text/plain") === -1
            ) {
              caches.open(version + cacheName).then(function (cache) {
                cache.put(request, copy);
              });
            }

            return response;
          })
          .catch(function () {
            // you can return an image placeholder here with
            if (request.headers.get("Accept")?.indexOf("image") !== -1) {
            }
          })
      );
    }) as Promise<Response>
  );
  return;
}

NOTE: If the request is for a plaintext resource it will be ignored by the service worker with this configuration. Additionally if request Accept header does not include "text/plain", but the response Content-Type header is of "text/plain" then the response will not be stored in the service worker cache. For any other response types like css, js, png, or jpg, the response will be cached by the service worker in this code section.

Here's what the entire service-worker.js file should look like will all the sections put together:

/// <reference lib="WebWorker" />

// export empty type because of tsc --isolatedModules flag
export type {};
declare const self: ServiceWorkerGlobalScope;

const cacheName = "::yourserviceworker";
const version = "v0.0.1";

self.addEventListener("install", function (event) {
  event.waitUntil(
    caches.open(version + cacheName).then(function (cache) {
      return cache.addAll(["/", "/offline"]);
    })
  );
});

self.addEventListener("activate", function (event) {
  event.waitUntil(
    caches.keys().then(function (keys) {
      // Remove caches whose name is no longer valid
      return Promise.all(
        keys
          .filter(function (key) {
            return key.indexOf(version) !== 0;
          })
          .map(function (key) {
            return caches.delete(key);
          })
      );
    })
  );
});

self.addEventListener("fetch", function (event) {
  const request = event.request;

  // Always fetch non-GET requests from the network
  if (request.method !== "GET") {
    event.respondWith(
      fetch(request).catch(function () {
        return caches.match("/offline");
      }) as Promise<Response>
    );
    return;
  }

  // For HTML requests, try the network first, fall back to the cache,
  // finally the offline page
  if (
    request.headers.get("Accept")?.indexOf("text/html") !== -1 &&
    request.url.startsWith(this.origin)
  ) {
    // The request is text/html, so respond by caching the
    // item or showing the /offline offline
    event.respondWith(
      fetch(request)
        .then(function (response) {
          // Stash a copy of this page in the cache
          const copy = response.clone();
          caches.open(version + cacheName).then(function (cache) {
            cache.put(request, copy);
          });
          return response;
        })
        .catch(function () {
          return caches.match(request).then(function (response) {
            // return the cache response or the /offline page.
            return response || caches.match("/offline");
          });
        }) as Promise<Response>
    );
    return;
  }

  // For non-HTML requests, look in the cache first, fall back to the network
  if (
    request.headers.get("Accept")?.indexOf("text/plain") === -1 &&
    request.url.startsWith(this.origin)
  ) {
    event.respondWith(
      caches.match(request).then(function (response) {
        return (
          response ||
          fetch(request)
            .then(function (response) {
              const copy = response.clone();

              if (
                copy.headers.get("Content-Type")?.indexOf("text/plain") === -1
              ) {
                caches.open(version + cacheName).then(function (cache) {
                  cache.put(request, copy);
                });
              }

              return response;
            })
            .catch(function () {
              // you can return an image placeholder here with
              if (request.headers.get("Accept")?.indexOf("image") !== -1) {
              }
            })
        );
      }) as Promise<Response>
    );
    return;
  }
});

Test Service Worker Locally

You can test the service worker by running your project locally with the http-server npm package. First make sure to compile the TypeScript service worker code by using the command npm run build-typecheck. Then, to install the http-server npm package run the command npm i http-server --save-dev. After installing run the command http-server in your project folder where the index.html and offline.html pages are. You should then see your website in the browser by navigating to the default http-server url localhost:8080. With browser dev tools you can inspect your website including managing service worker installation state and cache status.

In Chrome this is in the "Application" tab of chrome DevTools:

In the left hand navigation panel of the Application tab you can also see a section for "Cache Storage", and in the dropdown list should be an entry for the service worker, with the name you chose, listing all of the cached assets. There is three cached responses: the index.html page, the offline.html page, and the script.js file that registers the service worker.

Here's what that looks like in Chrome:

You can test offline mode by selecting the checkbox for "Offline" and deleting the cache item with name "/". Refreshing the page should load the offline.html since the index.html is no longer cached.

Your site can now support offline viewing for pages that are cached with the service worker, or show an offline page when there is no response previously cached and a network connection is unavailable.

Generate an XML Sitemap with Node.js

James Edwards — Tue, 09 Mar 2021 20:50:16 +0000

An xml sitemap informs search engines with information regarding the structure of a website and which pages should be available to be indexed in search results. The xml sitemap file includes the url location for all the pages included and the date the page was last modified. If you are building a blog website it is especially important to include a sitemap file containing information about all the blog posts. Recently more blogs are being built with the Jamstack, and since there is no server to dynamically serve the sitemap file, we can use Node.js to statically generate an xml sitemap file from blog post data. In this example we will use TypeScript along with the xml npm package to convert a JSON object containing blog post data into an xml string, and then write the generated xml string to a sitemap file. Before following these steps make sure to have Node.js and npm installed.

npm init package.json

The first thing we need to do is to generate a package.json file so that we can install the xml npm package. If you don't already have a package.json file setup for your project run the command npm init in the project folder and follow the prompts. Once the package.json file is created run the command npm install xml typescript --save. This will install the xml npm package and the TypeScript npm package. Since we are using TypeScript for this example we also need to install the type definitions for the xml package. These can be installed by running the command npm install @types/xml --save-dev.

Configure ES Module format

We are also going to use ECMAScript modules, or ES modules, instead of CommonJS modules, as Node.js now supports ES Module format. In order to use ES Modules the "type" property with value of "module" also needs to be added to the package.json file. Please read my other post for more information regarding how to import and export ES Modules in Node.js With these settings in place the package.json file should look similar to this:

{
  "type": "module",
  "name": "xmlsitemap",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "typescript": "^4.2.3",
    "xml": "^1.0.1"
  },
  "devDependencies": {
    "@types/xml": "^1.0.5"
  }
}

Configure TypeScript with Node.js

After adding the package.json file we can include the configuration steps that are needed to use TypeScript with ES Modules. To do this we can add a "tsconfig.json" file with the following settings:

{
  "compilerOptions": {
    "allowSyntheticDefaultImports": true,
    "isolatedModules": true,
    "strict": true,
    "module": "esnext",
    "lib": ["ES2019"],
    "moduleResolution": "node",
    "skipLibCheck": true
  },
  "include": ["index.ts"],
  "exclude": ["node_modules/**/*"]
}

The "module" setting with value "esnext" is the setting that configures the TypeScript compiler to compile TypeScript into JavaScript using the ES Module format.

Generate XML String with TypeScript

Now we can create the Node.js script that will generate the XML string that will be written to the sitemap file. To do this add a new file to the project named "index.ts". And add the following code:

import xml from "xml";
import { writeFile } from "fs";

async function main() {
  const pages = [
    {
      title: "Sample Page One",
      created: "Dec 22 2020",
      slug: "sample-page-one",
    },
    {
      title: "Sample Page Two",
      created: "Feb 1 2021",
      lastModified: "Feb 2 2021",
      slug: "sample-page-two",
    },
    {
      title: "Sample Page Three",
      created: "Mar 2 2021",
      lastModified: "Mar 5 2021",
      slug: "sample-page-three",
    },
    {
      title: "Sample Page Four",
      created: "Mar 20 2021",
      slug: "sample-page-four",
    },
  ];

  const indexItem = {
    //todo: build index item
  };

  const sitemapItems = pages.reduce(async function (
    items: { url: [{ loc: string }, { lastmod: string }] }[],
    item: {
      title: string;
      lastModified?: string;
      created: string;
      slug: string;
    }
  ) {
    // todo: build page items
    return items;
  }, []);
}

main();

This code is now set up to use sample data that is stored in the "pages" array. In this example we are including data to represent pages that would be included as part of a static site generator build process. Typically with a Jamstack blog this data would be sourced from markdown files, or another common option is requesting data from a headless content management system. For the purposes of this example we are including a short page list directly in the code, but usually this would be dynamically included at build time. After the sample page data there is one object that will contain the data for the sitemap index item, and the other is an array of objects containing the sitemap data for each individual page.

Create Sitemap Index Item

The first item in the sitemap will include optional tags that won't be included in the individual page sitemap items, and that is why it is created separately. Besides including the url location and a last modified time, the index sitemap item includes the change frequency parameter and a priority parameter. These are optional and can be included for each sitemap item, but in this case we are only including it for the root url of the sitemap. Go ahead an add the following inside of the "indexItem" object shown above:

const indexItem = {
  //build index item
  url: [
    {
      loc: "YOUR-DOMAIN-HERE",
    },
    {
      lastmod: new Date(
        Math.max.apply(
          null,
          pages.map((page) => {
            return (new Date(
              page.lastModified ?? page.created
            ) as unknown) as number;
          })
        )
      )
        .toISOString()
        .split("T")[0],
    },
    { changefreq: "daily" },
    { priority: "1.0" },
  ],
};

Make sure to replace "YOUR-DOMAIN-HERE" with your actual domain. Also note that in order to find the most recent date of all of the pages, the Math.max() function is used, in combination with the function prototype method .apply(), which passes the array of page object dates as parameters to the Math.max function. The first parameter of the .apply method is this, which is not needed so it is set to null.

Additionally since we are using TypeScript the date objects cannot be cast directly from a JavaScript Date object into a number, so they are cast to the unknown type as an intermediary step to prevent the TypeScript compiler from showing type errors. Once the maximum date of all the pages last modified or created dates is determined, it is formatted as an ISO string date format and then trimmed to only include the year, month, and day.

Create Sitemap Page Items

With the index sitemap item created, we can now build the individual page items in the "sitemapItems" array. This process will be similar to creating the index item, but each page item will only include a url location property and a last modified timestamp. To build the sitemap items add the following code to the sitemapItems array creation step:

const sitemapItems = pages.reduce(function (
  items: { url: [{ loc: string }, { lastmod: string }] }[],
  item: {
    title: string;
    lastModified?: string;
    created: string;
    slug: string;
  }
) {
  // build page items
  items.push({
    url: [
      {
        loc: `YOUR-DOMAIN-HERE/${item.slug}`,
      },
      {
        lastmod: new Date(item.lastModified ?? item.created)
          .toISOString()
          .split("T")[0],
      },
    ],
  });
  return items;
}, []);

For each page items url location property make sure to replace the placeholder text with your actual domain.

Build Sitemap Object

Now that both the sitemap index item and the sitemap items for each page are created we can combine them into one object that will be the entire sitemap object. At the end of the main function add in this code:

const sitemapObject = {
  urlset: [
    {
      _attr: {
        xmlns: "http://www.sitemaps.org/schemas/sitemap/0.9",
      },
    },
    indexItem,
    ...sitemapItems,
  ],
};

const sitemap = `<?xml version="1.0" encoding="UTF-8"?>${xml(sitemapObject)}`;

console.log(sitemap);

At this point we can test to make sure that the JSON to xml string conversion is working properly by logging the xml string to the console. To do this we need to add a script command to the package.json file created earlier.

Run Node.js Script with npm Scripts

In order to test the xml sitemap creation process we can add a script to the package.json file named "generate-sitemap". This script will invoke the TypeScript compiler and then run the transpiled JavaScript with Node.js. Here is what the script should look like in the package.json file:

{
  "scripts": {
    "generate-sitemap": "tsc && node index.js"
  }
}

We can run this script with the command npm run generate-sitemap. After running the generate sitemap command, the xml sitemap string should be output to the console. Here is what it will look like:

<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><url><loc>YOUR-DOMAIN-HERE</loc><lastmod>2021-03-20</lastmod><changefreq>daily</changefreq><priority>1.0</priority></url><url><loc>YOUR-DOMAIN-HERE/sample-page-one</loc><lastmod>2020-12-22</lastmod></url><url><loc>YOUR-DOMAIN-HERE/sample-page-two</loc><lastmod>2021-02-02</lastmod></url><url><loc>YOUR-DOMAIN-HERE/sample-page-three</loc><lastmod>2021-03-05</lastmod></url><url><loc>YOUR-DOMAIN-HERE/sample-page-four</loc><lastmod>2021-03-20</lastmod></url></urlset>

Instead of outputting the sitemap as an xml string we can write it to a file using the Node.js write file method in the fs module.

Write XML String to Sitemap File

You can replace the "console.log" statement at the bottom of the index.ts main function with the following code to write the sitemap xml string to a file named "sitemap.xml":

await writeFileAsync("./sitemap.xml", sitemap, "utf8");

You will also need to add one import statements at the top of the index.ts file. This will import the promisify function from the util module. This way we can convert the writeFile module to use promises instead of callbacks, which enables the use of async await syntax.

import { promisify } from "util";
const writeFileAsync = promisify(writeFile);

Here is what the entire index.ts file should look like with all the code included:

import xml from "xml";
import { writeFile } from "fs";
import { promisify } from "util";
const writeFileAsync = promisify(writeFile);

async function main() {
  const pages = [
    {
      title: "Sample Page One",
      created: "Dec 22 2020",
      slug: "sample-page-one",
    },
    {
      title: "Sample Page Two",
      created: "Feb 1 2021",
      lastModified: "Feb 2 2021",
      slug: "sample-page-two",
    },
    {
      title: "Sample Page Three",
      created: "Mar 2 2021",
      lastModified: "Mar 5 2021",
      slug: "sample-page-three",
    },
    {
      title: "Sample Page Four",
      created: "Mar 20 2021",
      slug: "sample-page-four",
    },
  ];

  const indexItem = {
    //build index item
    url: [
      {
        loc: "YOUR-DOMAIN-HERE",
      },
      {
        lastmod: new Date(
          Math.max.apply(
            null,
            pages.map((page) => {
              return (new Date(
                page.lastModified ?? page.created
              ) as unknown) as number;
            })
          )
        )
          .toISOString()
          .split("T")[0],
      },
      { changefreq: "daily" },
      { priority: "1.0" },
    ],
  };

  const sitemapItems = pages.reduce(function (
    items: { url: [{ loc: string }, { lastmod: string }] }[],
    item: {
      title: string;
      lastModified?: string;
      created: string;
      slug: string;
    }
  ) {
    // build page items
    items.push({
      url: [
        {
          loc: `YOUR-DOMAIN-HERE/${item.slug}`,
        },
        {
          lastmod: new Date(item.lastModified ?? item.created)
            .toISOString()
            .split("T")[0],
        },
      ],
    });
    return items;
  }, []);

  const sitemapObject = {
    urlset: [
      {
        _attr: {
          xmlns: "http://www.sitemaps.org/schemas/sitemap/0.9",
        },
      },
      indexItem,
      ...sitemapItems,
    ],
  };

  const sitemap = `<?xml version="1.0" encoding="UTF-8"?>${xml(sitemapObject)}`;

  await writeFileAsync("./sitemap.xml", sitemap, "utf8");
}

main();

You can then run the npm run generate-sitemap command again, and a new file named "sitemap.xml" should be created in the project folder. The contents of this file should be identical to the sitemap xml string that was logged to the console in the previous step.

Test Sitemap File in a Browser

To test out the sitemap in a browser, in the same project folder that we created the index.ts file run the command npm install http-server --save-dev, and add another script to the package.json file like this:

{
  "scripts": {
    "generate-sitemap": "tsc && node index.js",
    "serve": "http-server"
  }
}

Then to use the http-server npm package run the command npm run serve, and you should see the http-server npm package output the url it is serving. It is most likely the default setting so navigating to "localhost:8080/sitemap.xml" should show the sitemap file that will look similar to this:

Add Sitemap to robots.txt

Now you can include the sitemap generation step in the build process of the static site generator that you might be using for your Jamstack blog. You can also add a line to the robots.txt file to indicate the url of the sitemap file. If you are using a robots.txt file for your site make sure to add the following with your domain included:

Sitemap: https://YOUR-DOMAIN-HERE/sitemap.xml

Use the Dart Sass JavaScript Implementation to Compile SASS with Node.js

James Edwards — Thu, 25 Feb 2021 01:18:17 +0000

This post is an updated version of a previous post containing instructions on how to compile sass with the node-sass npm package, which is now deprecated.

The SASS team now recommends using Dart Sass in favor of LibSass for new development projects. This means that the sass npm package should be used instead of the node-sass npm package, which is built on top of LibSass, to compile sass with Node.js. The sass npm package is a pure JavaScript implementation of Dart Sass. The Dart Sass JavaScript API strives to be compatible with the existing node-sass API, so that it can be integrated into existing workflows with minimal changes. This post will show how to install the Dart Sass Javascript implementation with npm and use it via the supported JavaScript API and the command line. Before proceeding make sure to have Node.js and npm installed.

npm install sass

After installing Node.js and npm we can create a sample project to demonstrate the functionality of the sass npm package. To do this create a project folder and open it in a terminal window. Then run the command npm init and follow the prompts, which will create a package.json file. Then we can install the sass node module into the project, by running the command npm install sass --save.

We will also be using ES Module format for this example so the package.json requires an additional setting after generating. Add the "type" property to the package.json with the value set to "module", so that Node.js will use ES Modules rather than CommonJS modules. Here is some additional information about how to import and export ES Modules in Node.js, which explains why this setting is necessary.

Your package.json file should now look like this:

{
  "name": "npmsass",
  "type": "module",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "sass": "^1.32.8"
  }
}

SCSS

The sass npm package is now installed, but in order to use it we will need a SCSS file. In the same project folder create a new file named "styles.scss" and place the following code inside:

/* This CSS will print because %message-shared is extended. */
%message-shared {
  border: 1px solid #ccc;
  padding: 10px;
  color: #333;
}

// This CSS won't print because %equal-heights is never extended.
%equal-heights {
  display: flex;
  flex-wrap: wrap;
}

.message {
  @extend %message-shared;
}

.success {
  @extend %message-shared;
  border-color: green;
}

.error {
  @extend %message-shared;
  border-color: red;
}

.warning {
  @extend %message-shared;
  border-color: yellow;
}

The above SCSS code is borrowed from the Sass Basics guide and demonstrates one of the most useful features of Sass which is the @extend at-rule, to share a set of CSS properties among different selectors. Now that we have a SCSS file we can compile it to CSS using the sass npm package.

Compile Sass with Dart Sass JavaScript API

To use the sass npm package JavaScript API, we need to create the index.js file that is set to the "main" property value in the package.json file, created in the first step. This will be the entry point for the Node.js process that will carry out the SASS compilation. In the same project folder create a new file named "index.js", and add the following code:

import sass from "sass";
import { promisify } from "util";
const sassRenderPromise = promisify(sass.render);

async function main() {
  const styleResult = await sassRenderPromise({
    file: `${process.cwd()}/styles.scss`,
  });

  console.log(styleResult.css.toString());
}
main();

This code imports the sass package along with the util.promisify module and converts the sass render function to use promises instead of the default callback implementation. This makes working with the asynchronous API of the sass npm package more manageable because it allows for the use of async/await syntax.

After importing and "promisifying" the sass npm package, the main function contains the code to compile the styles.scss file into CSS. In order to run this code add the following the scripts property in the package.json file:

{
  "start": "node index.js"
}

We can then execute the main function by running the command npm run start, and the css output will be logged to the console.

Instead of logging directly to the console it is much more useful to write the CSS output to a file. The sass npm package does not expose a JavaScript API to write a file directly, however it does support a configuration property to indicate which file the CSS output will be written to. By adding this configuration and using the fs.writeFile module the CSS can be written to an output to a file like this:

import sass from "sass";
import { promisify } from "util";
import { writeFile } from "fs";
const sassRenderPromise = promisify(sass.render);
const writeFilePromise = promisify(writeFile);

async function main() {
  const styleResult = await sassRenderPromise({
    file: `${process.cwd()}/styles.scss`,
    outFile: `${process.cwd()}/styles.css`,
  });

  console.log(styleResult.css.toString());

  await writeFilePromise("styles.css", styleResult.css, "utf8");
}
main();

After running the npm run start command again, you should now see a styles.css file in the same project folder, that contains the compiled CSS output:

/* This CSS will print because %message-shared is extended. */
.warning,
.error,
.success,
.message {
  border: 1px solid #ccc;
  padding: 10px;
  color: #333;
}

.success {
  border-color: green;
}

.error {
  border-color: red;
}

.warning {
  border-color: yellow;
}

SASS Render Configuration Options

The sass npm package supports other render options including:

sourceMap
sourceMapContents
outputStyle

These can be added by modifying the options object passed into the sass render function. When including a source map file, a separate file needs to be written to the project folder containing the sourcemap information. To add these options make the following changes to the index.js:

import sass from "sass";
import { promisify } from "util";
import { writeFile } from "fs";
const sassRenderPromise = promisify(sass.render);
const writeFilePromise = promisify(writeFile);

async function main() {
  const styleResult = await sassRenderPromise({
    file: `${process.cwd()}/styles.scss`,
    outFile: `${process.cwd()}/styles.css`,
    sourceMap: true,
    sourceMapContents: true,
    outputStyle: "compressed",
  });

  console.log(styleResult.css.toString());

  await writeFilePromise("styles.css", styleResult.css, "utf8");

  await writeFilePromise("styles.css.map", styleResult.map, "utf8");
}
main();

Then run the npm run start command again and you should see the "styles.css" and "styles.css.map" files have both been updated.

The styles.css should now output with the blank spaces removed, and it will include a comment at the bottom to indicate the corresponding sourcemap file, which will look like this:

{
  "version": 3,
  "sourceRoot": "",
  "sources": ["styles.scss"],
  "names": [],
  "mappings": "AACA,kCACE,sBACA,aACA,WAaF,SAEE,mBAGF,OAEE,iBAGF,SAEE",
  "file": "styles.css",
  "sourcesContent": [
    "/* This CSS will print because %message-shared is extended. */\r\n%message-shared {\r\n  border: 1px solid #ccc;\r\n  padding: 10px;\r\n  color: #333;\r\n}\r\n\r\n// This CSS won't print because %equal-heights is never extended.\r\n%equal-heights {\r\n  display: flex;\r\n  flex-wrap: wrap;\r\n}\r\n\r\n.message {\r\n  @extend %message-shared;\r\n}\r\n\r\n.success {\r\n  @extend %message-shared;\r\n  border-color: green;\r\n}\r\n\r\n.error {\r\n  @extend %message-shared;\r\n  border-color: red;\r\n}\r\n\r\n.warning {\r\n  @extend %message-shared;\r\n  border-color: yellow;\r\n}\r\n"
  ]
}

The sourcemap will allow for easier debugging and the browser will now load both files. In the debug inspector the browser will show the line in the SCSS source code that corresponds to the CSS output being inspected.

Compile SASS using Dart Sass CLI

It is also possible to use the sass npm package directly from the command line. To do this with the same configuration as the example using the JavaScript API add the following the the package.json scripts property:

{
  "scripts": {
    "compileSass": "sass styles.scss styles.css --style=compressed --embed-sources"
  }
}

This will add a package.json script to run the SASS compiler, by running the command npm run compileSass. To make sure it is working as expected you might want to delete the previously generated styles.css and styles.css.map files, before running the npm run compileSass command.

Using the sass npm package JavaScript API or command line interface, should result in the same output consisting of both the css and css.map files, as both methods rely on the JavaScript implementation of Dart Sass. The main difference is that when using the CLI option the files will automatically be written based on the input and output specified, but when using the JavaScript API we must use the fs.writeFile module to write these files to the project folder.

Secure a Github Webhook with Node.js

James Edwards — Thu, 25 Feb 2021 01:11:28 +0000

GitHub provides webhooks that can send a POST request when a predetermined event is triggered. There are many different GitHub event types, and a common event to integrate into workflows is the PullRequestEvent. Any time a pull request has event activity of the following action types:

opened
closed
reopened
assigned
unassigned
review_requested
review_requested_removed
labeled
unlabeled
synchronize

a POST request can automatically be sent to trigger an integration that is waiting to accept the incoming request. In this example we can set up an Azure Serverless Function using Node.js to accept a GitHub Webhook POST payload. The serverless function will only run when the pull request is from the main branch, the branch associated with the pull request is merged, and the pull request is closed. If all of the following conditions are true, we will also make sure to secure the GitHub webhook by using the @octokit/webhooks npm package to verify the "x-hub-signature-256" request header using an application secret. When the incoming POST request payload is verified to be originating from GitHub's servers any application logic relating to the pull request closing, in the serverless function, can run as expected.

Set up Azure Serverless Function to Accept Webhook Post Request

The first thing we'll need to do is set up an Azure Serverless function so that there is an HTTP endpoint available to accept the incoming webhook POST request that will be sent from GitHub any time an event associated with a pull request occurs. It is not required to use Azure Serverless Functions with GitHub webhooks, so you can exchange this with another technology like a Node.js server using express. All that is required is an HTTP endpoint, using Node.js, that can accept incoming post requests.

Microsoft provides documentation for a quick start to create a function in Azure with TypeScript and Visual Studio code. This steps in this guide will build off that documentation so it is required to set that up before proceeding.

npm install @octokit/webhooks

Once you have the HTTP trigger function setup and you are able to run it locally as indicated in the quick start, we can add the @octokit/webhooks into the package.json that was automatically generated in the functions project. To do this use Visual Studio Code to open a terminal window in the folder where the package.json file was generated for the functions project. Then run the command npm install @octokit/webhooks --save. This will add the @octokit/webhooks npm package into the node_modules folder for the project, so that it can be imported into function code.

import @octokit/webhooks

In the HTTP Trigger function that was created by following quick start guide, this will be called "HTTPExample" if you did not change it, we need to add code to utilize the @octokit/webhooks package that was just installed. You can delete the sample code provided for "HTTPExample" function file named "index.ts". Then go ahead and add the following code into the index.ts file:

import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import { Webhooks } from "@octokit/webhooks";

const httpTrigger: AzureFunction = async function (
  context: Context,
  req: HttpRequest
): Promise<void> {
  context.log("HTTP trigger function processed a request.");

  context.res!.headers["Content-Type"] = "application/json";

  const payload = req.body;

  context.res!.status = 200;
  context.res!.body = { message: "success" };
};

export default httpTrigger;

This is the starting code needed to utilize the @octokit/webhooks npm package verify method. The code to do the verification hasn't been added, only the import statement that is on the second line of code. To use the verification method update the index.ts file to look like this:

import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import { Webhooks } from "@octokit/webhooks";

const httpTrigger: AzureFunction = async function (
  context: Context,
  req: HttpRequest
): Promise<void> {
  context.log("HTTP trigger function processed a request.");

  context.res!.headers["Content-Type"] = "application/json";

  // application/json post request body
  const payload = req.body;

  if (
    payload.action != "closed" ||
    payload.pull_request.base.ref != "main" ||
    !payload.pull_request.merged_at ||
    !new Webhooks({
      secret: process.env["GitHubWebhookSecret"],
    }).verify(payload, req.headers["x-hub-signature-256"])
  ) {
    // this pull request is either:
    //  not closed,
    //  not referencing the main branch,
    //  not merged,
    //  or is not valid
    // so by returning we are not going to process the application logic below
    return;
  }

  // your application logic goes here

  context.res!.status = 200;
  context.res!.body = { message: "success" };
};

export default httpTrigger;

Note: The "GitHubWebhookSecret" is not directly included in the code. Since this is a secret value it is more secure to access this as an environment variable. To add an environment variable within an Azure Functions project, you can view the documentation for how to add an application setting with the Azure portal. This value should be secret and not shared with anyone. In the upcoming steps we will add this to the GitHub repository webhook settings so that the @octokit/webhooks npm package can use this value to verify the request payload. If you are running your function locally you will also need to add the same "GitHubWebhookSecret" setting to the "local.settings.json" file that was automatically generated in the function project folder. Your local.settings.json file can include this value like this:

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "",
    "FUNCTIONS_WORKER_RUNTIME": "node",
    "GitHubWebhookSecret": "<YOUR-SECRET-VALUE>"
  },
  "Host": {
    "LocalHttpPort": 7071,
    "CORS": "*",
    "CORSCredentials": false
  }
}

With this code in place to secure the webhook, we can now be sure that any incoming requests are coming from GitHub's servers and meet the conditional criteria before processing further. Since the POST request from GitHub is sent when any event relating to a pull request occurs, the code above makes sure to only act on the incoming request if the pull request payload data indicates that the pull request is merged, closed, and from the main branch. This way if a pull request is opened or not associated to the main branch the webhook can ignore that request.

Configure GitHub Webhook Settings

Now that we can accept incoming POST requests with webhook payloads from GitHub we need to configure the repository settings to send the request when the pull request event occurs. To do this a GitHub repository will need to be created, if not existing already, and once created navigate to the "Settings" tab. Then in the secondary navigation for the repository settings there will be a nav item labelled "Webhooks". This is where we can configure the url for the webhook and the secret value that is used to verify the incoming request shown in the code above. You can click the button labelled "Add webhook" and GitHub will prompt you to enter your password to continue. Once you have entered your password you will see a screen like this:

To get the value for the Payload URL field, we need to enter the url for the function we created earlier. At this point if you have not deployed your Azure Serverless Function application, you can do this to get the url or, in the next section, follow the steps to set up ngrok to enable testing the locally running functions application.

If you want to deploy to Azure you will find the url in the function app overview settings panel in the Azure portal. This is only the function base url so you will need to append the route of the function that was created. If you kept the default function name the entire Payload URL field value will look something like this:

https://functionapp1.azurewebsites.net/api/httptrigger1

After the Payload URL field, the next field is for the Content type of the request that we are expecting from GitHub, in this case our function is set up to accept:

application/json

so make sure to update this setting. In the following field for the Webhook Secret provide the secret value that was saved as an application setting in the function app settings within the Azure portal. This will also be the same as the "secret" property value that was added to the "local.settings.json" file, shown earlier, within the functions project folder. Next we'll need to update the events that trigger the webhook so select the radio button for "Let me select individual events" then make sure to deselect the checkbox for "Pushes" and only select the checkbox for "Pull requests".

Then select Add webhook at the bottom and this will save the webhook configuration, and GitHub will automatically run a test request to make sure the webhook integration is working as expected.

Use ngrok to Test GitHub Webhook with Azure Functions Locally

If you don't want to deploy your Azure Severless Functions project to the Azure cloud while testing, you can use ngrok to test the GitHub webhook integration while running the function project locally. To do this download ngrok, and follow the installation instructions. Once it is setup, you can run the command ngrok http 7071 and ngrok will provide a publicly available url that will forward the port your functions app is running on. If you changed the default port for the functions project to something different than localhost:7071 make sure to run the ngrok command with the port you are using. With ngrok running you should get a url that looks like this:

http://92832de0.ngrok.io

With that url, go back to the GitHub webhook settings page in the repository settings and update the Payload URL field to match, ensuring that you have appended the entire url for the function so it would look like this:

http://92832de0.ngrok.io/api/httptrigger1

Then run your function app locally inside of Visual Studio Code, and save the webhook settings in GitHub. This will send another test request and you will be able to see the request, from GitHub's servers, being processed in the console output logs of your function app running locally.

Moving back to the "index.ts" file in your function project you can now add the code you need to integrate with the GitHub Webhook and it will use the @octokit/webhooks npm package to verify the incoming request was signed with the secret value you provided. Anytime an event occurs matching the webhook settings criteria, GitHub will send a POST request, and the webhook integration will occur automatically and securely. For an example of what can be done with GitHub webhooks, checkout out how to build a serverless comment system for a jamstack blog. Where you can find detailed instructions about how to setup a GitHub Webhook integration to provide a moderated commenting system for a blog without a database or servers.

Generate a Universally Unique Identifier (uuid) with Node.js

James Edwards — Tue, 23 Feb 2021 00:40:51 +0000

The uuid, or universally unique identifier, npm package is a secure way to generate cryptographically strong unique identifiers with Node.js that doesn't require a large amount of code. The uuid npm package has zero dependencies, and over thirty thousand packages depend on it, making it a safe choice when an id is needed that is guaranteed to be unique. It supports commonJS modules and also ECMAScript Modules, making it a good cross-platform choice. Besides generating a unique id, the uuid npm package has other utility methods available in its API that can be useful when working with unique identifiers, to ensure that they are valid.

npm install uuid

We can create a sample node.js script to test out the functionality of the uuid npm package, but first make sure that Node.js and npm are installed. Then run the command npm init in a test project folder terminal window, and follow the prompts that are shown. This will create a package.json file, which will be used to install the uuid package. Creating a package.json file is not needed, if you are adding uuid to an existing project.

After creating the package.json file run the command npm install uuid --save, and the uuid npm package will be added to the node_modules folder inside the project folder. If you are using TypeScript you can also run the command npm install @types/uuid to install the type definitions for the uuid npm package. See how to compile TypeScript with npm for more information if you want to use the uuid npm package with TypeScript.

Generate uuid with Node.js

With the uuid npm package installed we can now import it into a Node.js script and use the functions provided. Create a new file named "index.js" and include the uuid npm package like this:

import { v4 as uuidv4 } from "uuid";

function main() {
  const uniqueId = uuidv4();
  console.log(uniqueId);
}

main();

What this does is import the RFC4122 version 4 uuid export using ECMAScript modules syntax. If you aren't using ES Modules, you can by following the steps to import and export ES Modules in Node.js, or you can use commonJS modules with a different import syntax that looks like this:

const { v4: uuidv4 } = require("uuid");

For the remainder of this example we will use the ES Module syntax.

To test out the Node.js script, we can add a package.json script to run it. In the "scripts" property of the package.json file add the following:

{
  "scripts": {
    "start": "node index.js"
  }
}

After adding the additional script run the command npm run start, and you should see a uuid output to the console that looks similar to this:

d160b410-e6a8-4cbb-92c2-068112187503

You can re-run the command and a new uuid will be generated each time.

Validate uuid

The uuid npm package can also validate uuid strings to test whether they are a valid uuid. To do this add some additional code to the index.js file we just created.

import { v4 as uuidv4 } from "uuid";
import { validate as uuidValidate } from "uuid";

function main() {
  const uniqueId = uuidv4();
  console.log(uniqueId);

  const isValid = uuidValidate(uniqueId);
  console.log(isValid);
}

main();

Then run the npm run start command again and you will see the result of the uuidValidate method is output as true. If the value passed into the uuidValidate function is not a valid uuid the output will be false.

Detect uuid RFC version

After validation, if we have a valid uuid, the uuid npm package can also detect the version of a uuid string. Building off the index.js sample code add another import to access the uuid version function and test the uniqueId that is generated like this:

import { v4 as uuidv4 } from "uuid";
import { validate as uuidValidate } from "uuid";
import { version as uuidVersion } from "uuid";

function main() {
  const uniqueId = uuidv4();
  console.log(uniqueId);

  const isValid = uuidValidate(uniqueId);
  console.log(isValid);

  const version = uuidVersion(uniqueId);
  console.log(version);
}

main();

Now when we run the index.js script we can see the version of the uniqueId that is generated is output as "4", matching the uuidv4 version we imported initially. If the uuidVersion function is not passed a valid uuid an error will be thrown, so it can be helpful to wrap this function in a try/catch block to capture any errors.

import { v4 as uuidv4 } from "uuid";
import { validate as uuidValidate } from "uuid";
import { version as uuidVersion } from "uuid";

function main() {
  const uniqueId = uuidv4();
  console.log(uniqueId);

  const isValid = uuidValidate(uniqueId);
  console.log(isValid);

  try {
    const version = uuidVersion(uniqueId);
    console.log(version);
  } catch (error) {
    console.log(error);
  }
}

main();

This way any resulting error can be output to the console, or handled in a way that is best for the current usage.

Generate NIL uuid

If you need to generate a NIL uuid, or a uuid that is entirely zeros, the uuid npm package provides the NIL_UUID as an export. Adding it to the index.js sample script looks like this:

import { v4 as uuidv4 } from "uuid";
import { validate as uuidValidate } from "uuid";
import { version as uuidVersion } from "uuid";
import { NIL as NIL_UUID } from "uuid";

function main() {
  const uniqueId = uuidv4();
  console.log(uniqueId);

  const isValid = uuidValidate(uniqueId);
  console.log(isValid);

  try {
    const version = uuidVersion(uniqueId);
    console.log(version);
  } catch (error) {
    console.log(error);
  }

  console.log(NIL_UUID);
}

main();

I haven't come across a use case where a NIL uuid is needed, but it is provided in the uuid npm package so I'm sure there are plenty.

The uuid npm package is a helpful utility to generate a unique id with Node.js that saves a lot of the potential headache from generating one from scratch, while also ensuring the value are unique, secure, and matching a specific RFC version.

Import and Export ES Modules in Node.js using TypeScript with Babel Compilation

James Edwards — Mon, 15 Feb 2021 22:06:46 +0000

As of Node.js version 13.2.0 ECMAScript modules are now supported by default without adding an experimental flag. Although, using ES Modules without making the required configuration changes will result in the error "SyntaxError: Cannot use import statement outside a module". This is because Node.js, by default, is expecting the CommonJS module format.

Using TypeScript in combination with ES Modules brings many added benefits. To use TypeScript with ES Modules, the TypeScript compiler configuration in tsconfig.json can be updated to process code in ES Module format. Additionally, Babel can be used for TypeScript compilation, and the TypeScript compiler will be used for type checking, as Babel can not type check TypeScript code. Once the TypeScript code is being compiled by Babel into JavaScript, retaining the ES Module format, the ES Modules can be exported, imported, and run with Node.js.

package.json Type Module

The first configuration change we can make, to use ES Modules in Node.js is configuring the package.json file to include the type module property value. To do this add the following code to the package.json file in your Node.js project:

{
  "type": "module"
}

If you are starting a new project you can run the command npm init in a terminal window, follow the prompts that follow, and a package.json file will be generated in the current project folder. Although, before doing so make sure to have Node.js and npm installed. Once the package.json file is added to your project then add the extra configuration shown above as the npm init command does not generate a package.json file with this ES Module setting pre-configured.

npm install

We will also be using some additional npm packages to carry out the compilation and type checking processes.

Before proceeding run the command npm install cross-env @babel /cli @babel /core @babel /preset-env @babel /preset-typescript rimraf typescript --save. This will install the npm packages in the project "node_modules" folder and create a package-lock.json file. The npm packages are now available for usage in the project. Since we are using TypeScript, we can also run the command npm install @types/node --save-dev which will install the Node.js type definitions as a devDependency.

Configure TypeScript compiler to use ES Module format

Using ES Modules does not require the use of TypeScript, however the overhead of including TypeScript is minimal and including it provides many benefits such as static typing, which can enable code editors or an IDE to offer more predictive assistance. You may have heard referred to as intellisense or intelligent code completion. In the same folder as the package.json add a new file named "tsconfig.json" containing this configuration:

{
  "compilerOptions": {
    "allowSyntheticDefaultImports": true,
    "isolatedModules": true,
    "strict": true,
    "module": "esnext",
    "lib": ["ES2019"],
    "noEmit": true,
    "moduleResolution": "node",
    "skipLibCheck": true
  },
  "include": ["src/**/*.ts"],
  "exclude": ["node_modules/**/*", "dist/**/*"]
}

More info on tsconfig settings can be found in the TSConfig reference provided by Microsoft. The most important compiler option included is setting the "module" property to "esnext". This informs the TypeScript compiler to recognize source code in the ES Module format as well as retain the format when generating JavaScript code.

Since Babel will be configured to do the compilation of TypeScript into JavaScript the "noEmit" property is set to true, and what this does is allow for the use of the TypeScript compiler only to indicate when there are type checking errors. When configured this way the tsc compile command will not generate any JavaScript code, but it will output any errors that would occur during compilation to the console. It is also recommended, when using TypeScript with the Babel compiler, to set the "allowSyntheticDefaultImports" and "isolatedModules" to true as this ensures that the TypeScript compiler will process source code similar to how the Babel compiler does. This way the type checking and compilation configurations are in sync, even though separate steps are responsible for each.

Configure Babel to compile TypeScript into ES Modules

With TypeScript configured, we can add the Babel configuration that enables TypeScript compilation with the Babel compiler. To do this create a new file in the same folder as the tsconfig.json file named ".babelrc.json" and add this configuration:

{
  "presets": [
    ["@babel/preset-env", { "modules": false, "targets": { "node": true } }],
    ["@babel/preset-typescript"]
  ],
  "ignore": ["node_modules"],
  "comments": false,
  "minified": true
}

This will configure Babel to use the preset-typescript and preset-env when generating JavaScript code. The presets are executed in a bottom to top order, meaning that first Babel will compile the TypeScript into JavaScript and then on the resulting JavaScript code the preset-env configuration will be applied. This is where Babel is configured to use ES Modules as the "modules" setting is set to false, which is somewhat confusing because ES Modules are being used. It is necessary to set this to false otherwise Babel will use the default CommonJS module format for Node.js. Additionally the compilation target is set to Node so that Babel can apply code transforms that ensure the code will be able to run in the LTS version of Node.js.

In this example there are two extra babel settings included that instruct the Babel compiler to remove any comments in the source code and minify the JavaScript output. These can be removed if not desired for your use case, however this is beneficial for using in production to minimize code size.

Export ES Module

Now we can add some sample TypeScript code to test out the configuration changes.

In the same project folder create a new folders named "src", so that the file structure matches the "include" pattern in the tsconfig.json file. Then in the "src" folder create a new file named "helpers.ts" and place the following code in it:

function log(value: string) {
  console.log(value);
}

export { log };

This code is only logging the value that is passed in to the console, and is not really representative of actual code that would be used, but it allows for the demonstration of using ES Modules with TypeScript and Babel. The export of the "log" function is the key item to notice about this code, as this is all that is needed to export an ES Module. Now we can create another file to import the "log" helper function module.

Import ES Module

In the same "src" folder create a new file named "index.ts" this will be the main entry point for our ES Module code. Once that file is created add in this TypeScript code to import the helper function that was created in the previous step.

import { log } from "./helpers.js";

function main() {
  log("testing es modules");
}

main();

Similar to the helpers.ts file the index.ts files is mainly for demonstrating ES Module import syntax. It imports the helper function and then the main function is called to execute the "log" function. Although it is important to note that the file imported must end with a ".js" file extension rather than a ".ts" file extension. This is because when the code is eventually compiled the ES Module code will be a JavaScript file. Make sure that anytime a module is imported from a separate file the path is relative to the current file and the extension is set to ".js", otherwise both the TypeScript compiler and Babel compiler will not be able to resolve the file location.

Run ES Modules in Node.js

At this point the source code is configured to run with ES Modules, so we can now look at how to compile the code and run it with Node.js. To do this we'll need to add six additional scripts to the "scripts" property in the package.json file.

In the package.json "scripts" property add the following:

{
  "clean": "rimraf dist",
  "compile": "cross-env-shell babel src -d dist --source-maps --extensions '.ts'",
  "build": "npm run clean && npm run compile",
  "typecheck": "tsc --p .",
  "build-typecheck": "npm run typecheck && npm run build",
  "start": "npm run build-typecheck && node ./dist/index.js"
}

The "clean" script will ensure that prior to the compilation, the output directory "dist" will be deleted. This way the latest code will copied into an empty folder.

The "compile" script is where the cross-env package is used to run the babel compilation command. This babel compilation command specifies that the source files will be located in the "src" folder and when compilation is complete the JavaScript output will be copied to a folder named "dist". The flags that are passed in indicate that source maps should be generated for debugging purposes and also the "--extensions" flag is required so that Babel will look for files ending with the ".ts" extension.

To use the "clean" and "compile" script sequentially they are combined in a new script named "build", which can be run using the command npm run build. This will remove the old files from the "dist" folder and compile the TypeScript source code with Babel, however no typechecking errors will be indicated and Babel may fail to compile the code if there are errors present.

To resolve this an additional script "typecheck" is included that will pass the TypeScript source code through the TypeScript compiler, and if there are errors present, they will be output to the console. Since the tsconfig.json settings include the "noEmit" property the typecheck command won't output any JavaScript code.

The command that will be most commonly used is the "build-typecheck" command, which can be used by running npm run build-typecheck. This will sequentially run the "typecheck" command and then if there are no errors present as a result of the TypeScript compilation with the TypeScript compiler, the "build" command will be executed, invoking the Babel compiler and generating JavaScript code that can be run by Node.js in ES Module format.

Since the JavaScript code is being output to a folder named "dist" the "main" property in the package.json should be changed to:

{
  "main": "./dist/index.js"
}

To run the compiled JavaScript code, execute the command npm run start and this will carry out the type checking and compilation steps as well as run the index.js file with Node.js. If everything is setup and working as expected you should see the value included in the "main" function - "testing es modules" output to the console. Now you can use this configuration to create node modules that are statically typed and run in Node.js using the ES Module format.

Use Azure Table Storage with Azure Serverless Functions and Node.js

James Edwards — Mon, 01 Feb 2021 21:26:06 +0000

Before developing with Azure Serverless Functions and Azure Table storage locally, there are some tools required to emulate Azure Storage and provide a run-time environment for the Node.js serverless functions. Please make sure the prerequisites are set up before running the example code that follows.

Setup Azure Storage Emulator

In order to save on development costs, instead of creating cloud resources, we can install the Azure Storage Emulator for development and testing. If you aren't using windows, Azurite is an open source Azure storage API compatible server, and it is recommended by Microsoft to use. Otherwise, after installing, Windows users can search in the start menu for "azure storage emulator" and press enter to start the emulator. This should open a cmd window that will indicate the emulator is running, and some helpful commands. The cmd window can be closed and the emulator will continue to run.

Install Azure Storage Explorer

Next we'll need to download Azure Storage Explorer to interact with the emulated storage environment. This application is available for Windows, Mac, and Linux machines. After installing go ahead and start the Azure Storage Explorer, and in the left hand column navigator find the dropdown section labelled "Local & Attached" and then within that section find the secondary dropdown "Storage Accounts" and within the tertiary dropdown "(Emulator - Default Ports)" is where the resources, that we have not yet created, will be displayed. Here you can see three additional dropdown sections:

Blob containers
Queues
Tables

Our focus will be on the "Tables" section, which should be empty since no tables have been programmatically created yet.

Setup Azure Functions for Local Development

Now that we have the storage emulator and storage explorer configured we can download the Azure Functions extension for Visual Studio Code. If you don't have Visual Studio Code you can download it, and then follow the instructions to configure the local project. You don't need to follow the naming convention indicated in the documentation, but what is important is that there is a package.json created in the functions project. The package.json file is created automatically and allows us to include the npm package provided by Microsoft to interact with Azure Storage.

npm Install azure-storage

In the same folder as the package.json that was created, run the command npm install azure-storage --save and then run the command npm install @types/node --save-dev to install the type definitions for Node.js. This will install the azure-storage npm package to the local functions project so that we can import it in our code.

Congratulations, you made it through the setup configuration!

Http Trigger Serverless Function

Now we can write the code to use Azure Table Storage with Azure Serverless Typescript Functions. To begin find the file "index.ts" in the HTTP trigger function that was created earlier (if using the default it will be called HttpTrigger1). Right now there is sample code in that function that can be deleted, and the code below can be added.

The function should now look like this:

import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import * as azureStorage from "azure-storage";

const httpTrigger: AzureFunction = async function (
  context: Context,
  req: HttpRequest
): Promise<void> {
  context.log("HTTP trigger function processed a request.");

  // set content type for all responses
  context.res!.headers["Content-Type"] = "application/json";

  if (req.method == "POST") {
  } else if (req.method == "GET") {
  } else if (req.method == "PUT") {
  } else if (req.method == "DELETE") {
  } else {
    // request method does not match
    context.res!.status = 500;
  }
};

export default httpTrigger;

Programmatically Create Table If Not Exists

Before we can retrieve data from Azure Storage we need to insert data by using an HTTP POST request, additionally a table must be created to store the data. To ensure there is a table to store data we can programmatically create the entity table if it does not exist with the azure storage npm package, at the time of the POST request. In order to connect to the storage emulator a connection string is required, which can be stored as an environment variable to be passed into the Node.js serverless functions process. To do this add the default local connection string to the file "local.settings.json" that is in the same folder as the HttpTrigger1 function. Additionally we want to add a "Host" configuration to permit CORS requests and set the default port that the functions will run on.

The local.settings.json file should now look like this:

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "",
    "FUNCTIONS_WORKER_RUNTIME": "node",
    "TableStorageConnection": "UseDevelopmentStorage=true"
  },
  "Host": {
    "LocalHttpPort": 7071,
    "CORS": "*",
    "CORSCredentials": false
  }
}

Now we can use the "TableStorageConnection" environment variable to create a table.

import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import * as azureStorage from "azure-storage";

const httpTrigger: AzureFunction = async function (
  context: Context,
  req: HttpRequest
): Promise<void> {
  context.log("HTTP trigger function processed a request.");

  const tableService = azureStorage.createTableService(
    process.env["TableStorageConnection"]
  );

  const createTableIfNotExists = (tableName: string) =>
    new Promise((resolve, reject) => {
      tableService.createTableIfNotExists(tableName, (error, result) => {
        if (error) {
          reject(error);
        } else {
          resolve(result);
        }
      });
    });

  // set content type for all responses
  context.res!.headers["Content-Type"] = "application/json";

  if (req.method == "POST") {
    try {
      await createTableIfNotExists("TestTable");
    } catch (error) {
      console.log(error);
      context.res!.status = 400;
      context.res!.body = {
        message: "An error occurred.",
      };
    }
  } else if (req.method == "GET") {
  } else if (req.method == "PUT") {
  } else if (req.method == "DELETE") {
  } else {
    // request method does not match
    context.res!.status = 500;
  }
};

export default httpTrigger;

Once that table service is initialized the "tableService.createTableIfNotExists" function can be used. This function by default, uses a callback function to obtain the result. Instead of using the callback, the function is wrapped in a Promise which can asynchronously resolve the callback function or return an error. Following that the promise is awaited inside a conditional branch that will only execute if the incoming request is a POST request.

The function can now create a new table if it doesn't exist named "TestTable" on any incoming POST request. To test this out run the function (in Visual Studio Code press F5), and then download Postman to mock requests. Copy the url provided in the terminal window where the function is running, if you kept the default configuration this url will be "http://localhost:7071/api/HttpTrigger1", and change the request method in Postman from GET to POST and send the request. In the response body displayed in Postman all that will show is the number "1", however if we use the Azure Storage Explorer to view the emulator tables we can see that the "TestTable" was successfully created. You may need to select "refresh all" in the storage explorer to see the new table.

Insert Azure Table Storage Entity

Now that the table will be programmatically created if it doesn't exist, we can add a request body to the POST request that is being sent in Postman. This data will be parsed with the querystring module included with Node.js and then a storage entity can be generated from the incoming data. Once the storage entity is generated it can then be saved to the storage table.

To facilitate the saving of the table data we can use the uuid npm package, to install run the command npm install uuid --save and then install the typescript type definitions with the command npm install @types/uuid --save-dev.

Add the following import statements to the index.ts file:

import * as querystring from "querystring";
import { v4 as uuidv4 } from "uuid";

Then add the following inside the POST method conditional branch:

//parses www-form-urlencoded request body
const body = querystring.parse(req.body) as {
  firstName: string;
  lastName: string;
  age: string;
};

if (!(body && body.firstName && body.lastName && !isNaN(Number(body.age)))) {
  context.res!.status = 400;
  context.res!.body = {
    message: "The data is invalid.",
  };
  return;
}

// inform table storage of row types
const entityGenerator = azureStorage.TableUtilities.entityGenerator;

// storing data within the same storage partition
// partition key and row key combo must be unique but also type string
const entityData = {
  PartitionKey: entityGenerator.String("TestPartition"),
  RowKey: entityGenerator.String(uuidv4()),
  firstName: entityGenerator.String(body.firstName),
  lastName: entityGenerator.String(body.lastName),
  age: entityGenerator.Int32(body.age),
};

try {
  const tableName = "TestTable";

  await createTableIfNotExists(tableName);

  const entity = await insertEntity(tableName, entityData);

  context.res!.status = 200;
  context.res!.body = {
    message: "Data is saved.",
    data: entity,
  };
} catch (error) {
  console.log(error);

  context.res!.status = 400;
  context.res!.body = {
    message: "An error occurred.",
  };
}

Note: Azure Table Storage requires both the partition key and the row key value to be present on storage entities and it also enforces that the type of these columns is a string. The "RowKey" property is utilizing the uuid package that was installed to guarantee that the partition key and row key combination is unique regardless of the other entity data. It's also worth noting that the entity generator isn't required and Azure Table Storage will default to a type of string if the entity row type is not specified.

You will notice that there is no function declared yet with the name "insertEntity". We can add that helper function below the "createTableIfNotExists" function.

const insertEntity = (tableName: string, entity: {}) =>
  new Promise((resolve, reject) => {
    tableService.insertEntity(tableName, entity, (error, result) => {
      if (error) {
        reject(error);
      } else {
        resolve(result);
      }
    });
  });

After adding the code to save the table storage entity run the serverless functions again with visual studio code, and submit a post request containing sample data with Postman.

Here is what the request should look like in Postman:

Checking with the Azure Storage Explorer, inside of the "TestTable" there should be one entity, and now we can add code to retrieve this data entity using the partition key and row key values that are saved.

Retrieve Azure Storage Entity

In order to retrieve the Azure Storage entity we will need to add a second helper function to the index.ts file inside the HttpTrigger1 serverless function. This helper function will allow us to retrieve storage entities using the partition key and the row key. Below the "insertEntity" function declaration add the following code:

const retrieveEntity = (
  tableName: string,
  partitionKey: string,
  rowKey: string
) =>
  new Promise((resolve, reject) => {
    tableService.retrieveEntity(
      tableName,
      partitionKey,
      rowKey,
      (error, result) => {
        if (error) {
          reject(error);
        } else {
          resolve(result);
        }
      }
    );
  });

Then the "retrieveEntity" helper function can be called in the conditional branch that will execute on incoming GET requests, however we will need a way to pass the row key value to the function from the incoming request data. To do this we can customize the http endpoint using the functions.json file that is in the HttpTrigger1 function (the same folder as index.ts). In that file add a new key to the first object in the "bindings" array.

The functions.json file should look similar to this :

{
  "bindings": [
    {
      "authLevel": "anonymous",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req",
      "methods": ["get", "post"],
      "route": "HttpTrigger1/{rowKey:guid?}"
    },
    {
      "type": "http",
      "direction": "out",
      "name": "res"
    }
  ],
  "scriptFile": "../dist/HttpTrigger1/index.js"
}

In the route parameter that is being added the pattern to match the row key in the request is specified. The row key will be of type GUID and is optional since post requests will not have a row key to specify. We can now use the retrieve entity function in combination with the request parameter to query Azure Table Storage for the entity data.

Add this code into the GET request method branch to retrieve and return the data:

try {
  const entity = await retrieveEntity(
    "TestTable",
    "TestPartition",
    context.bindingData.rowKey
  );

  context.res!.status = 200;
  context.res!.body = {
    message: "Data retrieved.",
    data: entity,
  };
} catch (error) {
  console.log(error);
  context.res!.status = 400;
  context.res!.body = {
    message: "An error occurred",
  };
}

Then in Postman change the request method to GET and copy the entity row key from table storage so that the url in Postman looks similar to

http://localhost:7071/api/HttpTrigger1/99baf118-fb0b-495e-b839-432264ff6aaa

The row key will be different in your case since it is automatically generated for each entity, so make sure to change that to the entity row key saved to your local table storage. In the response data from postman you should see the following data returned:

{
  "message": "Data retrieved.",
  "data": {
    "PartitionKey": {
      "$": "Edm.String",
      "_": "TestPartition"
    },
    "RowKey": {
      "$": "Edm.String",
      "_": "99baf118-fb0b-495e-b839-432264ff6aaa"
    },
    "Timestamp": {
      "$": "Edm.DateTime",
      "_": "2021-01-30T20:51:49.323Z"
    },
    "firstName": {
      "_": "test first"
    },
    "lastName": {
      "_": "test last"
    },
    "age": {
      "_": 99
    },
    ".metadata": {
      "metadata": "http://127.0.0.1:10002/devstoreaccount1/$metadata#TestTable/@Element",
      "etag": "W/\"datetime'2021-01-30T20%3A51%3A49.323Z'\""
    }
  }
}

In the data property of the response, each of the table storage columns is returned as an object containing two properties, one indicating the table storage data type and the other is the value of the property. There is also an additional metadata field included in the response that provides extra info about the response from Azure Table Storage, or in this case the Azure Storage Emulator.

Azure table storage entities can now be inserted and retrieved, but it is also useful to be able to update an entity that has been previously saved. To do this we can add the PUT request method to the "methods" property of the first object in the "bindings" array located in the functions.json file. Update the "methods" property to look like this:

{
  "methods": ["get", "post", "put", "delete"]
}

The code for the delete method is going to be added later, so that string value has also been added to the array at this time.

Update Azure Storage Entity

After permitting PUT requests in the functions.json add this code to the PUT method conditional branch:

//parses www-form-urlencoded request body
const body = querystring.parse(req.body) as {
  rowKey: string;
  firstName: string;
  lastName: string;
  age: string;
};

// inform table storage of row types
const entityGenerator = azureStorage.TableUtilities.entityGenerator;

// use request body data to maintain row key for entity
const entityData = {
  PartitionKey: entityGenerator.String("TestPartition"),
  RowKey: entityGenerator.String(body.rowKey),
  firstName: entityGenerator.String(body.firstName),
  lastName: entityGenerator.String(body.lastName),
  age: entityGenerator.Int32(body.age),
};

try {
  const entity = await updateEntity("TestTable", entityData);

  context.res!.status = 200;
  context.res!.body = {
    message: "Data is updated.",
    data: entity,
  };
} catch (error) {
  console.log(error);
  context.res!.status = 400;
  context.res!.body = {
    message: "An error occurred",
  };
}

A third helper function is also needed, shown as "updateEntity" so it can be added below the "retrieveEntity" helper function, above the request method conditional branches:

const updateEntity = (tableName: string, entity: {}) =>
  new Promise((resolve, reject) => {
    tableService.replaceEntity(tableName, entity, (error, result) => {
      if (error) {
        reject(error);
      } else {
        resolve(result);
      }
    });
  });

The "updateEntity" function takes two parameters one being the table name and the other is the updated entity. The partition key and the row key of the entity must match an existing partition/row key combination, or table storage will return an error. If desired there is a function provided by the azure-storage npm package named "insertOrReplaceEntity" which, as the name indicates can either update existing entities or create a new one if one does not exist. In this example the entity already exists so only the "replaceEntity" function is needed.

The PUT request method branch to update an existing entity is almost the same as the POST method branch to insert a new storage entity. The url is the same for both, and the main difference is that the "rowKey" is included in the request body so that the appropriate entity can have it's data updated. You can try it out by changing one of the fields in the request body to a different value and then check in the storage explorer to confirm the entity that was previously inserted and retrieved has the matching table column value updated.

Delete Azure Storage Entity

Much like the GET request method branch the DELETE request method does not contain a request body, instead the row key will be passed in the request as a parameter, and like the examples above we can add a fourth helper function to carry out the deletion.

const deleteEntity = (tableName: string, entity: {}) =>
  new Promise((resolve, reject) => {
    tableService.deleteEntity(tableName, entity, (error, result) => {
      if (error) {
        reject(error);
      } else {
        resolve(result);
      }
    });
  });

Then use the "deleteEntity" function in the DELETE request method branch by adding this code:

try {
  // inform table storage of row types
  const entityGenerator = azureStorage.TableUtilities.entityGenerator;

  // use request body data to maintain row key for entity
  const entityData = {
    PartitionKey: entityGenerator.String("TestPartition"),
    RowKey: entityGenerator.String(context.bindingData.rowKey),
  };

  const statusMessage = await deleteEntity("TestTable", entityData);

  context.res!.status = 200;
  context.res!.body = {
    message: "Data deleted.",
    data: statusMessage,
  };
} catch (error) {
  console.log(error);
  context.res!.status = 400;
  context.res!.body = {
    message: "An error occurred",
  };
}

To test this copy the row key value from the storage explorer for the entity previously saved and use the same url from the GET request method example in Postman, but change the request method to DELETE. Then execute the delete request with Postman and in the response section a success message will be displayed:

{
  "message": "Data deleted.",
  "data": {
    "isSuccessful": true,
    "statusCode": 204,
    "body": "",
    "headers": {
      "cache-control": "no-cache",
      "content-length": "0",
      "server": "Windows-Azure-Table/1.0 Microsoft-HTTPAPI/2.0",
      "x-ms-request-id": "3c378130-7a6d-4652-9022-d02320d29c05",
      "x-ms-version": "2018-03-28",
      "x-content-type-options": "nosniff",
      "date": "Sun, 31 Jan 2021 21:23:06 GMT"
    }
  }
}

The response status from Azure Table Storage is 204 No Content, since there is no longer an entity saved in the table. We can verify the entity was deleted by refreshing the table in the storage explorer. The response items shown in the "statusMessage" variable, is the response from Azure Table Storage, and it is being included in the response back from the serverless function to show the consumer of the serverless function API that the delete request to Azure Storage was successful. If the delete request failed the status message would indicate that by setting the "isSuccessful" property value to false.

Here is the complete function file with all code include:

import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import * as azureStorage from "azure-storage";
import * as querystring from "querystring";
import { v4 as uuidv4 } from "uuid";

const httpTrigger: AzureFunction = async function (
  context: Context,
  req: HttpRequest
): Promise<void> {
  context.log("HTTP trigger function processed a request.");

  const tableService = azureStorage.createTableService(
    process.env["TableStorageConnection"]
  );

  const createTableIfNotExists = (tableName: string) =>
    new Promise((resolve, reject) => {
      tableService.createTableIfNotExists(tableName, (error, result) => {
        if (error) {
          reject(error);
        } else {
          resolve(result);
        }
      });
    });

  const insertEntity = (tableName: string, entity: {}) =>
    new Promise((resolve, reject) => {
      tableService.insertEntity(tableName, entity, (error, result) => {
        if (error) {
          reject(error);
        } else {
          resolve(result);
        }
      });
    });

  const retrieveEntity = (
    tableName: string,
    partitionKey: string,
    rowKey: string
  ) =>
    new Promise((resolve, reject) => {
      tableService.retrieveEntity(
        tableName,
        partitionKey,
        rowKey,
        (error, result) => {
          if (error) {
            reject(error);
          } else {
            resolve(result);
          }
        }
      );
    });

  const updateEntity = (tableName: string, entity: {}) =>
    new Promise((resolve, reject) => {
      tableService.replaceEntity(tableName, entity, (error, result) => {
        if (error) {
          reject(error);
        } else {
          resolve(result);
        }
      });
    });

  const deleteEntity = (tableName: string, entity: {}) =>
    new Promise((resolve, reject) => {
      tableService.deleteEntity(tableName, entity, (error, result) => {
        if (error) {
          reject(error);
        } else {
          resolve(result);
        }
      });
    });

  // set content type for all responses
  context.res!.headers["Content-Type"] = "application/json";

  if (req.method == "POST") {
    //parses www-form-urlencoded request body
    const body = querystring.parse(req.body) as {
      firstName: string;
      lastName: string;
      age: string;
    };

    if (
      !(body && body.firstName && body.lastName && !isNaN(Number(body.age)))
    ) {
      context.res!.status = 400;
      context.res!.body = {
        message: "The data is invalid.",
      };
      return;
    }

    // inform table storage of row types
    const entityGenerator = azureStorage.TableUtilities.entityGenerator;

    // storing data within the same storage partition
    // partition key and row key combo must be unique but also type string
    const entityData = {
      PartitionKey: entityGenerator.String("TestPartition"),
      RowKey: entityGenerator.String(uuidv4()),
      firstName: entityGenerator.String(body.firstName),
      lastName: entityGenerator.String(body.lastName),
      age: entityGenerator.Int32(body.age),
    };

    try {
      const tableName = "TestTable";

      await createTableIfNotExists(tableName);

      await insertEntity(tableName, entityData);

      context.res!.status = 200;
      context.res!.body = {
        message: "Data is saved.",
        data: entityData,
      };
    } catch (error) {
      console.log(error);
      context.res!.status = 400;
      context.res!.body = {
        message: "An error occurred.",
      };
    }
  } else if (req.method == "GET") {
    try {
      const entity = await retrieveEntity(
        "TestTable",
        "TestPartition",
        context.bindingData.rowKey
      );
      context.res!.status = 200;
      context.res!.body = {
        message: "Data retrieved.",
        data: entity,
      };
    } catch (error) {
      console.log(error);
      context.res!.status = 400;
      context.res!.body = {
        message: "An error occurred",
      };
    }
  } else if (req.method == "PUT") {
    //parses www-form-urlencoded request body
    const body = querystring.parse(req.body) as {
      rowKey: string;
      firstName: string;
      lastName: string;
      age: string;
    };

    // inform table storage of row types
    const entityGenerator = azureStorage.TableUtilities.entityGenerator;

    // use request body data to maintain row key for entity
    const entityData = {
      PartitionKey: entityGenerator.String("TestPartition"),
      RowKey: entityGenerator.String(body.rowKey),
      firstName: entityGenerator.String(body.firstName),
      lastName: entityGenerator.String(body.lastName),
      age: entityGenerator.Int32(body.age),
    };

    try {
      const entity = await updateEntity("TestTable", entityData);
      context.res!.status = 200;
      context.res!.body = {
        message: "Data is updated.",
        data: entity,
      };
    } catch (error) {
      console.log(error);
      context.res!.status = 400;
      context.res!.body = {
        message: "An error occurred",
      };
    }
  } else if (req.method == "DELETE") {
    try {
      // inform table storage of row types
      const entityGenerator = azureStorage.TableUtilities.entityGenerator;

      // use request body data to maintain row key for entity
      const entityData = {
        PartitionKey: entityGenerator.String("TestPartition"),
        RowKey: entityGenerator.String(context.bindingData.rowKey),
      };

      const statusMessage = await deleteEntity("TestTable", entityData);

      context.res!.status = 200;
      context.res!.body = {
        message: "Data deleted.",
        data: statusMessage,
      };
    } catch (error) {
      console.log(error);
      context.res!.status = 400;
      context.res!.body = {
        message: "An error occurred",
      };
    }
  } else {
    // method does not match any
    context.res!.status = 500;
  }
};

export default httpTrigger;

Azure serverless functions are a scalable and cost efficient method to create a RESTful API interface to interact with Azure Table Storage. The code above covers the actions needed to insert, retrieve, update and delete Azure Storage entities, while using TypeScript and the azure-storage and uuid npm packages to execute the methods that correspond to the Azure Table Storage API. Azure Storage entities can be accessed from a consumer of the serverless functions REST API, like a web application, and the Azure Storage credentials and connection string remain secure.

Split a TypeScript Array Into Chunks With a Generic Reduce Method

James Edwards — Mon, 01 Feb 2021 21:19:13 +0000

Running too many asynchronous processes simultaneously with Node.js can cause issues that will lead to the process crashing. An example of this is when reading files inside of an asynchronous callback function that is being executed using the map() method on an array. To prevent a scenario where the node.js process might crash with an EMFILE error, it can be helpful to split an array into smaller arrays or chunks, and process the group of smaller arrays synchronously while asynchronously mapping over the items in each of the smaller arrays. By doing this the contents of the original array can be processed in batches, preventing an error caused by opening too many files at once in parallel. The following configuration will allow us to demonstrate the EMFILE error and then add code to split an array into chunks, batching the process, and preventing the error from occurring.

Setup Node.js and npm package.json

Make sure to have node.js and npm installed before following these steps. Then run the command npm init and follow the prompts to create a package.json file. Once the package.json file is created add the setting:

{
  "type": "module"
}

This will permit the use of ECMAScript modules in the code, specifically it will allow for the use of es module imports from npm packages. After that we need to install TypeScript, so run the command npm install typescript --save and then run the command npm install @types/node --save-dev. At this point also go ahead and add a new script property called "start", that will initiate the TypeScript compiler and run the JavaScript output with Node.js.

The package.json file should look similar to this:

{
  "type": "module",
  "name": "splitarrayintochunks",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "tsc && node index.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "@types/node": "^14.14.22"
  },
  "dependencies": {
    "typescript": "^4.1.3"
  }
}

Setup TypeScript

After configuring Node.js, add a tsconfig.json file to same folder as the package.json file. This lets us use TypeScript, which we just installed, instead of JavaScript and as a result we get the advantage of generic types among other features. Copy this config into the tsconfig.json file:

{
  "compilerOptions": {
    "allowSyntheticDefaultImports": true,
    "isolatedModules": true,
    "strict": true,
    "module": "esnext",
    "lib": ["ES2019"],
    "moduleResolution": "node",
    "skipLibCheck": true
  },
  "include": ["*.ts"],
  "exclude": ["node_modules/**/*"]
}

Now the output of the TypeScript compilation, indicated in the tsconfig "module" field, will be created as ECMAScript modules, which matches the type field added to the package.json configuration.

Node.js EMFILE Error When Reading Files

The configuration steps are now complete and we can add some code that will demonstrate the EMFILE error that can be prevented by batch processing the array in smaller chunks. This sample code, that results in an error, can be added to index.ts.

import fs from "fs";
import util from "util";
const readFile = util.promisify(fs.readFile);

(async function main() {
  //an array containing ten thousand undefined items
  const originalArray = Array.from(Array(10000));

  try {
    // awaiting all ten thousand promises simultaneously
    await Promise.all(
      originalArray.map(async () => {
        const file = await readFile("./data.json", "utf8");
        console.log(file);
      })
    );
  } catch (error) {
    console.log(error);
  }
})();

At this point also create a sample JSON file referenced in the code above named "data.json". All that you need to add to this file is "{}" which will be interpreted as an empty JSON object. With the data file created run the command npm run start and as expected you should see an error in the console:

[Error: EMFILE: too many open files, open '/../../data.json'] {
  errno: -4066,
  code: 'EMFILE',
  syscall: 'open',
  path: '/../../data.json'
}

What is occurring is that we are trying to asynchronously read the data.json file ten thousand times all at once, and the error is informing us that there are too many file descriptors for the system that the code is being run on. The access to the data.json file is happening too frequently for the system to keep track of and as a result the process crashes.

Rather than trying all ten thousand file read attempts at once, the array can be split into chunks and the read requests can be processed in batches, ensuring the number total number of file descriptors is within a suitable limit for the system that Node.js is operating on. To do this we can create a generic TypeScript function that will split any type of array into chunks of the original array type.

TypeScript Generic Reducer to Split Array Into Chunks

In the index.ts file, and above the main function that is immediately invoked we can create another function named "chunkItems". This will utilize TypeScript generics to create an array containing groups of smaller arrays, that match the type of the original array.

const chunkItems = <T>(items: T[]) =>
  items.reduce((chunks: T[][], item: T, index) => {
    const chunk = Math.floor(index / 512);
    chunks[chunk] = ([] as T[]).concat(chunks[chunk] || [], item);
    return chunks;
  }, []);

The reduce() method is used to create an array containing chunks of smaller arrays, and for this example the chunk size is set to be a limit of 512 items per chunk. This way the maximum number of file descriptors that can be allocated at once, is below the default limit of most systems. Now we can use the generic "chunkItems" function to create a batched process by wrapping the existing file read code in a for...of loop, so that each of the Promise.all() results can be awaited asynchronously.

Putting all the code together in the index.ts file looks like this:

import fs from "fs";
import util from "util";
const readFile = util.promisify(fs.readFile);

const chunkItems = <T>(items: T[]) =>
  items.reduce((chunks: T[][], item: T, index) => {
    const chunk = Math.floor(index / 512);
    chunks[chunk] = ([] as T[]).concat(chunks[chunk] || [], item);
    return chunks;
  }, []);

(async function main() {
  const originalArray = Array.from(Array(10000));
  const chunks = chunkItems(originalArray);
  try {
    for (const chunk of chunks)
      await Promise.all(
        chunk.map(async (item, index) => {
          const file = await readFile("./data.json", "utf8");
          console.log("-----start item------");
          console.log("current array chunk:" + chunks.indexOf(chunk));
          console.log("file contents: " + file);
          console.log("current item index: " + index);
          console.log("-----end item-------");
        })
      );
  } catch (error) {
    console.log(error);
  }
})();

Run the npm run start command again, and the EMFILE error will not occur. The output of the above code will display rather quickly, but it will show the index of each chunk currently being processed synchronously and the contents of the sample data.json file. Watching closely (or by stopping the output after it has ran for sometime), you can see that the chunk index goes in numerical order, but the intentionally limited number of file reads is still happening asynchronously and the current item indexes are not in numerical order. By splitting the array into smaller chunks the system is not overloaded and Node.js is able to process the files asynchronously.