Leveraging Node Modules to Provide Offline Documentation

Are you a young, hip, developer-on-the-go? Neither am I, but I do tend to do some application development in situations where I don’t have access to the internet. Also I’m not hip. And youth is fleeting.

Anyway.

There are very good solutions for obtaining offline-documentation. DevDocs comes to mind, because it offers great offline documentation storage for lots of well-used software, and it has tons of features. In fact, you should probably just use that. However, you can’t just add offline documentation for any package you want. There is some gatekeeping that occurs.

If an existing solution like DevDocs doesn’t suit your needs, you can leverage the documentation that ships with the packages you’ve installed in your application. That’s right, I’m talking about README.md files.

Over the weekend ™️, I had the idea of building a little CLI tool that could create an express server that would look for the node_modules directory and serve up the contents of each packages’ README.md file. The tool would also provide a web interface for you to search node_modules for packages. It could also uses IndexedDB to store favorites offline.

So I did it. The result is Module Docs and you can install it as a Node package.

You can install it globally, or per project. After installing, start the cli by running:

$ module-docs start

You can create an npm script that will automatically start module-docs as part of the dev process. Here’s how I use it:

{
  "scripts:" {
    "start": "npm run start:docs && webpack-dev-server",
    "start:docs": "module-docs start"
  }
}

You can configure module-docs for each project you want to use it on by creating a module-docs.config.js file in the root of that projects’ directory. Currently, you can provide an array of package names to include as favorites, like so:

// module-docs.config.js
module.exports = {
   favorites: ["react", "react-apollo", "react-apollo-hooks"]
}

If you just want to use it, you can stop reading here and go live your best life. If you want to read on about how it was built, bless up and keep reading.

Creating the CLI

In order to create the cli, I’ll use commander.js, which is a very popular CLI building tool.

const program = require("commander")
const makeServer = require("./server/serve")
const path = require("path")

// gets the config file from the working directory of the application 
const getConfig = () => {
  const configPath = path.join(process.cwd(), "./module-docs.config.js")
  const config = require(configPath)
  return config ? config || null
}

// using commander, execute the start command which spins up the express server
program.command("start").action(() => {
  const modulePath = path.join(process.cwd(), "./node_modules")
  const config = getConfig()
  makeServer(modulePath, config)
})

program.parse(process.argv)

This is the jumping off point for the entire module_docs package. Its what allows you to run module-docs start to start up the express server. Let’s take a look at the server.

Building The Server

The server is a pretty basic Node server build using Express. It uses webpack-dev-middleware to create a dev server that will serve up a React application for the web UI.

const express = require("express")
const webpack = require("webpack")
const config = require("../webpack.config")
const devMiddleware = require("webpack-dev-middleware")
const compiler = webpack(config)
const bodyParser = require("body-parser")

// controller to handle API requests
const FileController = require("./controllers")

// Probably should make this configurable 
const PORT = 4444


module.exports = (modulePath, config) => {
  const app = express()

  app.use(bodyParser.json())

  // start webpack dev server
  app.use(
    devMiddleware(compiler, {
      open: true,
      stats: "errors-only"
    })
  )

  // handles getting package names from node_modules
  app.post("/modules", FileController.getFiles(modulePath, config))

  // handles getting the package info and README from a package
  app.post("/module/:name", FileController.getPackage(modulePath))

  app.get("*", function response(req, res) {
    res.sendFile("./client/template.html", { root: __dirname })
  })

  app.listen(PORT, () => {
    console.log(`Module Docs is running at http://localhost:${PORT}`)
  })
}

As you can see, there are two API endpoints. The first endpoint handles getting the directory names from node_modules. The second endpoint gets the README contents and parses package.json for information about the package. Currently, the UI just displays the package version and a link to the package’s homepage, if there is one.

To handle the POST request, I’ve created a FileController. This is where all the heavy lifting is.

The FileController

This file could definitely use some refactoring. That being said, I’ll break this file down into chunks. First, the utility functions and imports:

const fs = require("fs")
const pipe = require("lodash/fp/pipe")
const some = require("lodash/some")
const promisify = require("util").promisify
const readdir = promisify(fs.readdir)
const readFile = promisify(fs.readFile)

// directories to exclude from the search
const blacklist = [".bin", ".cache", ".yarn-integrity"]

const filterThroughBlacklist = files =>
  files.filter(f => !blacklist.includes(f))

// check to see if the list of files includes the filename
const checkFilesForFile = files => fileName =>
  some(files, f => f.name === fileName)

// Get all the files in the package that are directories. This is used
// for mono-repos are scoped packages that don't contain README files directly. 
// I could probably refactor this and the blackListFilter into one util function
const getDirectories = files =>
  files.filter(f => f.isDirectory() && f.name !== "node_modules")

// checks a package directory to see if it contains a README or a package.json file
const checkPackage = files => {
  const checkFilesFor = checkFilesForFile(files)
  return {
    hasReadme: checkFilesFor("README.md"),
    hasPackageInfo: checkFilesFor("package.json")
  }
}

// gets the content of the README and the package.json file, if they exist
const getDirectoryContent = async directory => {
  const files = await readdir(directory, { withFileTypes: true })
  const { hasReadme, hasPackageInfo } = checkPackage(files)
  const readmeContent =
    hasReadme && (await readFile(`${directory}/README.md`, "utf8"))

  const packageInfo =
    hasPackageInfo && (await readFile(`${directory}/package.json`, "utf8"))

  return {
    files,
    readmeContent,
    packageInfo
  }
}

// If a package has sub-directories, check each directory for a README and package.json
// If they exists, get contents of each and return
const getPackagesFromChildren = parentDir => children => {
  const readmes = children.map(async child => {
    const childDir = `${parentDir}/${child.name}`
    const { readmeContent, packageInfo } = await getDirectoryContent(childDir)
    return readmeContent || packageInfo
      ? {
          name: child.name,
          path: `${childDir}/README.md`,
          content: readmeContent,
          info: packageInfo
        }
      : {}
  })

  return Promise.all(readmes)
}

It’s important to note that I’ve created a blacklist of files to exclude from searching, because they are in node_modules but aren’t useful for our purposes. I’m sure this list is not all encompassing.

Also, we should provide a list of sub-directories (children) that contain README and package.json files, in case the package in question is a mono-repo, or a scoped package, like babel. This is what getPackagesFromChildren does.

The above util functions will be used in the FileController handlers. Let's take a look at them.

// gets directory names from a path, excluding blacklisted names. Returns an array of strings.
exports.getFiles = (path, config) => async (req, res) => {
  const files = await readdir(path)
  const filteredFiles = filterThroughBlacklist(files)
  res.send({ files: filteredFiles, config })
}

// Gets README content for package and all first-level children.
exports.getPackage = path => async (req, res) => {
  const name = req.params.name
  const dir = `${path}/${name}`

  try {
    const { files, readmeContent, packageInfo } = await getDirectoryContent(dir)
    const children = await pipe(
      getDirectories,
      getPackagesFromChildren(dir)
    )(files)

    const pkg = {
      path: dir,
      content: readmeContent,
      info: packageInfo,
      children: children
    }

    res.send({ pkg })
  } catch (err) {
    console.log("Unable to scan directory: " + err)
    res.send({ pkg: "No Readme Found" })
  }
}

That’s pretty much it for the server-side of things.

As for the front-end, its built using React (as of writing 16.8-alpha, so I can use those sweet sweet hooks). You're best bet is to play around with the CodeSandbox below.

Wrap Up

Not bad for a weekend. I’m going to keep working on some other features that I’d personally like to use, like being able to add custom notes to each package, and being able to link and fetch documentation and saving that offline as well. Let me know what you think!