DEV Community: Hugh Rawlinson

How to Ship man pages with your Node Programs

Hugh Rawlinson — Sat, 14 Aug 2021 12:07:38 +0000

This post was first published on my website.

The Javascript ecosystem uses the npm package registry to distribute libraries - and also binaries. If you've ever done npm install -g yarn, you've used this feature. The yarn package exposes a binary (a node js script) that executes on your system as a program, often invoked in the command line. Your npm installation then installs that package in a central global directory, and links each binary exposed by the package to a central directory that's in your shell. Then you can invoke the binary on your command line.

hugh@hugh-XPS-13-9343 ~> yarn --version
1.22.10

Pretty neat right?

In unix systems, the man utility is a common way to look up how to use a given command.

hugh@hugh-XPS-13-9343 ~> man yarn
No manual entry for yarn

But not all packages provide man pages. This isn't terrible - in the case of yarn, there is a whole 'help' subcommand to look up information about how to use yarn's cli - plus a documentation website. But, if like me, you think the developer experience is improved by meeting your developer where they expect you to be, you might like to distribute at least a nearly empty man page pointing devs in the right direction. If there's some chance of some portion of your users reaching for man {your binary}, I would suggest that it's worth weighing the effort of providing at least a man page with the benefit that those users would get from having docs (or a pointer to docs) where they expect.

Isn't that effort huge? Aren't man pages in a weird format for fancy native developers? How would I even install them from an npm package? In this post I'll attempt to convince you that making a basic man page is not a huge lift, and might be worth the your work for the developer experience benefit.

Let's start off with a creating a basic empty package for demonstration purposes, and installing some dependencies that will help us create our man pages.

I've created an example repo for you to refer to if you like.

$ mkdir my-package && cd my-package
$ npm init --yes
$ npm install --save marked<1 marked-man

Now we find ourselves with an empty package with two dependencies - marked, and marked-man. marked is a peer dependency of marked-man, the package that will take our markdown document and convert it to roff, the format used by man-pages. If you'd like to cut down dependencies and write in roff directly, go ahead! But I figure most javascript developers will be more familiar with Markdown.

Right now, there's a bug in marked-man causing it not to support versions of it's peerDependency marked greater than 1.0.0. At the moment, I suggest installing a version of marked below 1, and keeping an eye on the bug.

Next, let's write a simple document. Store the following in README.md.

# my-package(1) -- demo package

## Synopsis

my-package is a demonstration package that does nothing

## See also

example.com

There's a git more going on here than simple markdown syntax. Let's walk through it. On the first line, we have a heading containing our package name, followed immediately by a number in brackets. This number refers to the man 'section number' for your page. In our case, we're using section number 1 to show that our documentation concerns "Executable programs or shell commands" - but you can specify library calls, special files, games, etc. Check out the table in the man man document. The package name(section number) is followed by a -- spacer and a short description of your command.

Later on in the document, we can see sections headed by h2s. These are man "section names". You can have custom sections, but conventional section names include NAME (which is automatically generated for you), SYNOPSIS, CONFIGURATION, DESCRIPTION, FILES, NOTES, BUGS, AUTHORS, SEE ALSO, and more that are shown in the man man document.

Time to use marked-man to generate the man file. Man files are stored in the roff format, and we can use marked-man as follows to generate our roff file from our README.md.

$ ./node_modules/.bin/marked-man README.md

You'll see the following output:

.TH "MY\-PACKAGE" "1" "June 2021" "" ""
.SH "NAME"
\fBmy-package\fR \- demo package
.SH Synopsis
.P
my\-package is a demonstration package that does nothing
.SH See Also
.P
example\.com

I'm certainly glad I don't have to manually write in that format! Let's store this in a directory, and add an npm script so that we don't have to type out the full command every time. Add the following entry to your 'scripts' object in your package.json.

    "generate-man-page": "mkdir -p man && marked-man README.md > ./man/my-package.1"

Note that the roff output is stored in the man directory in a file called my-package.1. For your man file, you should follow the same naming convention: {packageName}.{section number}.

How do we make the man utility aware of the document when we install the package? We add an entry pointing at the file in our package.json as follows:

  "man": ["./man/my-package.1"]

We can test this out by running "npm install --global .", which will install the package in the current directory onto the system globally. Then, run man my-package to see the man page in action.

MY-PACKAGE(1)                                                                        MY-PACKAGE(1)

NAME
       my-package - demo package

Synopsis
       my-package is a demonstration package that does nothing

See Also
       example.com

                                          June 2021                                  MY-PACKAGE(1)

There we have it! Man pages for your node packages. If you'd like to have multiple man pages for your package (say your command is configurable by a dotfile that you'd like to document, for example), you can write multiple markdown documents, modify your npm script to generate them all, and add them to the list of exports on your package.json man object.

You may notice in the npm documentation that there's a directories.man configuration in the package.json spec, which is documented as exporting all the man pages to the system. I wasn't able to get that to work. If you are, please let me know!

Thanks for reading. I hope that I've covered the procedure for generating man pages from markdown documents such that it's clear to you - and I hope you agree that it's a relatively low amount of effort to add a touch of delight to your package's developer experience. If you have any feedback, please get in touch on twitter or on mastodon. I'd love to hear from you!

What to do if you publish a beta build as @latest

Hugh Rawlinson — Sat, 26 Jun 2021 19:30:51 +0000

I recently published a beta build of Meyda to the npm registry, with the intention of having one of our longest running users test it out to make sure it worked in their project. I hadn't done a manual release in a long time, since we use semantic-release, so I skimmed the output of npm publish --help, and figured out what command I would run. I
set the version field of package.json to 5.1.7-beta.0, as instructed built the bundle, ran our test suite, and ran npm publish . --dry-run, to verify that the manifest of files that would be published was correct. It was correct, and so I ran

npm publish .

When I checked Meyda's page on npm, I was rather surprised to see that 5.1.7-beta.0 had been published as the latest tagged version of Meyda. I had incorrectly been assuming that the magic incantation required to publish a beta package was the -beta.* suffix in the package version. In fact, the way to publish a beta version of an npm package is

npm publish . --tag beta

At this point, I became worried. Had I published a beta build of a package that might inadvertently contained breaking changes to all my users? While yes, technically I had done that (for the second time that week, but that's another story), I needn't have worried. Some research revealed that while packages once published to the registry in most cases cannot be removed or modified, tags can. The npm dist-tag command saved the day! I'll leave you to read the npm dist-tag --help, and instead show what I did to resolve my situation. The previous "good" latest version of my package was 5.1.7.

# Tag the previous version as latest
npm dist-tag add meyda@5.1.7 latest

# Tag the beta as a beta
npm dist-tag add meyda@5.1.8-beta.0 beta

Once the tags were properly set, none of our users were at risk of getting broken code they hadn't opted in for, and the user who agreed to test our beta release was able to install it with npm install meyda@beta.

No need to worry if you find yourself in this situation. Like most things, it's completely recoverable, and everything will be ok. We all lived happily ever after!

Loading Audio in Node JS

Hugh Rawlinson — Tue, 01 Jun 2021 18:49:27 +0000

Working with audio as a developer can unlock many awesome features, and a lot of fun. You can generate music, analyze audio using machine learning, build audio visualizers, music information retrieval systems, and much more. It's an extremely fun field. But working with audio can be tricky - how is sound represented on a computer? How can we manipulate that sound? And how do we serialize sound data to disk?

Pulse Code Modulation Encoding

This post won't be a deep dive into audio encoding - it's a practical guide for how to load audio in Node JS, into a state that you can work with it. Generally, digital signal processing (which means "working with audio data using code") operates on a kind of audio data called Pulse Code Modulation. There's a lot of fancy theory and maths behind PCM encoding - but until you're ready to dive in to Wikipedia, you can think of it as "a long list of numbers that represent the change in air pressure over time that makes up a sound". This is, after all, what a microphone measures and converts into numbers.

Samples

Each number in the list that makes up a sound is called a "sample". The sample can be represented on disk as one of several kinds of numbers - floating point numbers, integers, or other representations. The number of bits that represent the number affect the precision of the number - for example, 16 bit numbers can have much more precision than 8 bit numbers. The number of bits in each sample is referred to as the "bit depth".

Sample Rate

Another important attribute of PCM encoded audio is the "sample rate". This refers to the rate at which samples should be played in order for the sound to be at the right speed. For reasons outside the scope of this post, the sample rate dictates the highest frequency component that can be represented in a sound. For the purposes of most audio intended for human listening, it's important to store audio at a sample rate slightly higher than double the maximum frequencies that humans can hear. Since humans can't really hear audio over 20,000hz, a standard sample rate has emerged at 44,100hz. The "hz" unit here refers to hertz, which means "samples per second". Sometimes you can encounter audio with a higher or lower sample frequency - audio for movies can be up to 192,000hz, and signals representing things that aren't meant for human hearing (for example, geological sonar scans) might not need as many as 44,100 samples per second.

Loading PCM audio from disk

Several audio file formats store PCM encoded audio directly - wav and aiff are examples.

Luckily, other developers have implemented great libraries that handle the complexities of parsing wav files for you. I recommend node-wav, by Andreas Gal. It's got a simple API, and uses the metadata at the start of the wav file to automatically choose the correct sample rate, bit depth, and number encoding. From the readme, here is a code example.

let fs = require('fs');
let wav = require('node-wav');

let buffer = fs.readFileSync('file.wav');
let result = wav.decode(buffer);
console.log(result.sampleRate);
console.log(result.channelData); // array of Float32Arrays

The result.channelData variable contains a list of signals that you can use as standard Javascript Float32Arrays. The result object also exposes the sample rate, which you will likely need to know for many operations.

If you're using Meyda to analyze audio that you load in this way, you will need to make sure that the sample rate of the audio matches the sample rate that Meyda is set to use. Otherwise you'll end up with audio features that are incorrect, and based on a skewed frequency scale. You can either match the Meyda sample rate to the wav sample rate, or you can resample the audio to fit a standard sample rate (i.e. 44,100hz, or 48,000hz). Resampling audio is a complicated topic beyond the scope of this article, but if you have trouble finding information online, let me know and I may find time to write an article.

AIFF files also store PCM audio data, but differ from WAV files in that they have a different header format for storing metadata. node-wav doesn't support AIFF files, and I haven't found a package I would recommend to do so. If you need to analyze AIFF files, I would suggest using a utility like ffmpeg to transcode the audio to wav.

What about non-PCM audio formats?

But what about audio file formats like mp3, ogg, and flac? The difference between these formats and wav is that the audio is compressed on disk. mp3 and ogg are what's called "lossy" compression - that means they change the actual sound in ways that are hopefully imperceptible to most listeners in order to get better compression. flac, meanwhile, is a format that implements lossless compression. This means that it encodes audio on disk in a more efficient format than storing each sample as a full integer or floating point number, but without modifying the audio itself.

Encoding agnostic signal processing code

It's best to write signal processing code that works with one representation of audio, and reuse it by converting the audio - rather than having one implementation of your signal processing code for each audio encoding. We can achieve code reusability by converting all audio to a common format for signal processing, so that your code only has to think about one representation. Libraries that do this are called "codecs" which comes from "enCOding/DECoding". In order to support a particular file format in your program, you will need to make sure that you have the right codec. Luckily, you don't need to understand each audio format and implement a codec yourself - you can use packages to do this. So when you're writing your signal processing code, you should write code that works on raw signals, not encoded or compressed. In many cases, in Javascript, signals are represented as Float32Arrays - and unless you have specific requirements where this causes a limitation for you, I would recommend sticking to writing code that assumes signals are in Float32Arrays.

Loading alternative encodings from disk

While there are some implementations of mp3 encoders in Javascript, I would actually recommend calling out to another technology to do the transcoding. ffmpeg is a long running open source project that excels in media encoding. It can translate between many different media encodings, and I'm confident that it covers a huge portion of transcoding needs. In Node, you can call out to ffmpeg using the child_process API.

import { exec } from "child_process";
import { mkdtemp } from "fs/promises";
import path from "path";
import os from "os";

// Create a temporary directory to store transcoded audio
const TEMP_DIR = await mkdtemp(path.join(os.tmpdir(), "transcoder-storage-"));

async function transcodeToWav(filename) {
  return new Promise((resolve, reject) => {
    let output_filename = `${path.join(TEMP_DIR, filename)}.wav`;
    // "shell out" to ffmpeg
    exec(
      `ffmpeg -i ${filename} ${output_filename}`,
      (error, stdout, stderr) => {
        if (error) {
          console.log("ERROR: ", error);
          reject(error);
        }
        resolve({ filename: output_filename, stdout, stderr });
      }
    );
  });
}

try {
  let result = await transcodeToWav("./164064__cclaretc__rooster.mp3");
  // result.filename is the new filename of the transcoded audio.
  // We can now use node-wav as described above to read the audio

  let buffer = fs.readFileSync('file.wav');
  let decodedAudio = wav.decode(buffer);
  console.log(decodedAudio.sampleRate);
  console.log(decodedAudio.channelData); // array of Float32Arrays
} catch {}

I'm using a modern version of Nodejs which allows imports, top level await in .mjs files, and exposes the fs/promises interface, but this code refactors back to older versions of node if you need.

One thing to bear in mind is that in order for this to work, you need to have a copy of ffmpeg on the system that you're running the code on. Luckily, there's a package for that - ffmpeg-static is a dependency that you can include in your project that installs a statically linked copy of ffmpeg. You can use it to ensure that ffmpeg is always available to your code. Check it out!

But what about the web?

While in theory it might be possible to run ffmpeg through emscripten and run it in a web worker (I certainly assume someone has done this), it's not necessarily practical to try and use the same technique from node to transcode audio on the web. The good news is that the w3c has chartered a working group to focus on web codecs. While this is at the time of writing still in early stages, the working group is powering ahead on designing and proposing an API to enable media transcoding on the web, and hopefully that will become available to us in the near future.

What did we learn?

In this blog post, I covered the basics of Pulse Code Modulation encoding, how to load wav files from disk, the difference between wav files and other audio encoding file formats, transcoding other file formats to wav for loading in node, and how transcoding might soon work outside of node, but on the web. I hope these explanations have been useful to you. If anything is unclear, or you have more questions, please let me know on Twitter! Thanks for reading.

Handling Pagination with Async Iterators

Hugh Rawlinson — Thu, 20 May 2021 16:36:16 +0000

When you're interacting with a server from your frontend Javascript code, you might need to handle paging. Paging is a technique used by API designers to avoid enormous (and sometimes impossibly large) responses to requests when providing access to large collections of information to clients. Instead of returning every single item in a collection as a response to a request, an API might return the first 50 of the items in the collection, and a message to the client to say "this isn't all the items in the collection. If you want to get the next 50 items, here's how".

That's what the Spotify API does. When you need to get a list of albums by particularly prolific performers, you won't necessarily be able to get them all in one page, and will have to handle pagination to get all the albums.

It's possible to interact with pagination in an imperative way.

let artistId = '6sFIWsNpZYqfjUpaCgueju';

async function loadAlbums(artistId, authToken) {
  let endpoint = `https://api.spotify.com/v1/artists/${artistId}/albums?limit=20&include_groups=album`;

  let albums = [];
  // We'll set endpoint to page.next when we receive it in the response.
  // When there is no more data, the API will set page.next to null, and we'll
  // escape this while loop.
  while (endpoint) {
    const response = await fetch(endpoint, {
      headers: {
        "Authorization": `Bearer ${authToken}`
      }
    });

    if (!response.ok) {
      throw new Error("Request failed");
    }

    const page = await response.json();

    albums = albums.concat(page.items);

    endpoint = page.next;
  }
  return albums;
}

for (let album of (await loadAlbums(artistId, YOUR_OWN_AUTH_TOKEN))) {
  console.log(album.name);
}

This code works, but there are some problems with it.

The code that is consuming the data is mixed with the code that handles pagination.

You can extract the code that handles the pagination by converting the whole block into an async function. But since functions can only return data once, you're stuck until all the requests are finished before you can return albums and use it.

This is where async generators come in. Generators are functions that can yield multiple results, rather than just one. Asynchronous (async) generators are analogous to Promises that can resolve multiple times. They also provide syntactic sugar to make it easier to iterate over the yielded values - for await ... of syntax.

Async Iterators are one solution to this problem - observables are another solution, but they haven't made it into the EcmaScript specification.

The following is some example code that demonstrates how to use a recursive async generator to yield each page of albums one by one until we're out of pages. You'll see how the code that consumes the albums uses the for await ... of syntax to access the results of the generator

async function* pageThroughResource(endpoint, authToken) {
  async function* makeRequest(_endpoint) {
    const response = await fetch(_endpoint, {
      "headers": {
        "Authorization": `Bearer ${authToken}`
      }
    });
    if (!response.ok) {
      throw new Error(await response.text());
    }

    const page = await response.json()

    yield page;

    if (page.next) {
      yield * makeRequest(page.next);
    }
  }

  yield * makeRequest(endpoint);
}

async function* loadAlbums(artistId, authToken) {
  const endpoint = `https://api.spotify.com/v1/artists/${artistId}/albums?limit=20&include_groups=album`
  const result = pageThroughResource(endpoint, authToken);

  for await (const page of result) {
    for (let album of page.items) {
      yield album;
    }
  }
}

for await (const album of loadAlbums("6sFIWsNpZYqfjUpaCgueju", YOUR_OWN_AUTH_TOKEN)) {
  console.log(album.name);
}

In this example, the code that's responsible for making requests to the paginated external service is abstract - the behavior responsible for managing the pagination (the pageThroughResource function) doesn't know about what it's paginating through. The logic that knows about loading albums (the loadAlbums) function is what handles the specific details of the API that we're calling. The only assumption that the pageThroughResource function makes is that the response object from the API returns a field called next which provides the URL of the next page of the resource listing. This means that you can re-use the pageThroughResource function on any API call you need to make that has the same pagination design.

The code achieves the separation of these two distinct behaviors by creating functions that return asynchronous iterators. pageThroughResource returns an asynchronous iterator, but also internally defines another function, makeRequest, that also returns an asynchronous iterator. pageThroughResource uses the yield * syntax to yield to whatever makeRequest's resulting async iterator returns. The code is organized this way so that makeRequest is able to call itself recursively. Inside makeRequest, first the JSON result of the response of the API call is yielded, and the user can use it immediately. After that, only if the response contains a next field, makeRequest will delegate control of the generator to another instance of itself, made to handle the next page. While that request is being made, the calling code already has access to the result of the first page. That means we don't have to wait until all the pages are loaded before we can start using the information we get from the API.

These specific functions make a few assumptions, including:

the API you're calling will return JSON
the JSON that your API returns will contain a field called next, which provides the next page of the resource listing for you to call

But you can use this pattern in your own code, tailored to however your API handles response types and pagination data. You could even use this pattern to page through a resource in a GraphQL API.

One specific drawback to point out: iterators in Javascript don't have the map, reduce, and filter algorithms that you might know from arrays - you'll have to use the for await .. of syntax to handle their output. Maybe one day we'll get that interface!

I hope this helps you keep your code nice and maintainable!