Read All Files of Directory and Subdirectories with Recursive Generators in JavaScript

Reading all files from a directory and its subdirectories is a common requirement in many projects. While there are several ways to accomplish this, one approach is to use a recursive generator function to create a sequence of file paths.

Let's start by taking a look at the code:

import fs from 'fs';
import path from 'path';

export function* readAllFiles(dir: string): Generator<string> {
  const files = fs.readdirSync(dir, { withFileTypes: true });

  for (const file of files) {
    if (file.isDirectory()) {
      yield* readAllFiles(path.join(dir, file.name));
    } else {
      yield path.join(dir, file.name);
    }
  }
}

The function readAllFiles takes a directory path as input and returns a generator object that creates a sequence of file paths.

Inside the function, we first use fs.readdirSync to read all files and directories in the specified directory. We pass the withFileTypes option to get fs.Dirent objects instead of filenames.

Then we loop through each file or directory with a for..of loop. If the current item is a directory, we recursively call the readAllFiles function with the new directory path using the yield* syntax. This creates a nested sequence of file paths for each subdirectory.

If the current item is a file, we use the path.join function to construct the full path of the file and yield it.

Now that we have our generator function, let's see how we can use it to read all the files in a directory and its subdirectories.

for (const file of readAllFiles('/path/to/directory')) {
  console.log(file);
}

We can simply use a for..of loop to iterate through the generator object and print each file path to the console.

Using a recursive generator function to read all the files in a directory and its subdirectories has several benefits over other approaches. Instead of pushing all file paths into an array, the generator yields each path one at a time. This means that only one file path is kept in memory at a time, making it more memory efficient. Moreover, the generator only generates file paths when they are requested by the consuming code. This means that if we only need to process a subset of the files, we don't waste time and resources generating paths for files we don't need.

Comments

[Nik]

Interesting approach. Really like it!

[Robert]

I'm been out of coding for about twelve years. I guess this means I'm a novice. I'm trying to use this to list files and directories on a portable flash drive. Would you be so kind as to provide a html/javasript example?

[Chris Cook]

Sure, I can help you out. Are you using Node.js?

[Robert]

Well.... I tried Node.js but was thoroughly confused by it. Do you link all the scripts or just Node.js? If so, does Node.js link/call from the other files?

[Chris Cook]

I think running this kind of script in a browser is not a good idea, because the browser has only limited access to the file system. That's more a job for a command line script written in JavaScript but executed in a JS runtime like Node.js or Bun.

Here is an example that read all files and sub-directores start with directory test1: Sandbox

You can copy the index.js file to a local file and then run node index from the command line in the directory where this file is stored. You must change the pathToDir to the path of your flash drive.

Let me know if I need help with anything!