Tricks on writing & appending to a file in Node.js

This article covers the use of fs.appendFile and fs.writeFile functions, how they work in details. Specifically, we'll investigate them in a practical case.

Writing logs

Let's discover a use case where we want to write logs to a file. It seems like there is an obvious way to do this - call fs.writeFile each time we need it.

fs.writeFile('log.txt', 'message', 'utf8', callback);

The problem is writeFile replaces the file data each time we use the function, so we can't just write to a file. We could use a different approach: read a file data via fs.readFile, then append to the existing logs a necessary data and new line.

// we'll use callbacks in the article, but remember you always
//  can promisify those functions
// *we will not handle the errors in callbacks
const newLogs = `${Date.now()}: new logs`;
fs.readFile('log.txt', { encoding: 'utf8' }, (err, data) => {
  const newData = data + newLogs + '\n';
  fs.writeFile('log.txt', newData, 'utf8', callback);
});

But this method also has cons. Each time we want to write new logs, the program opens a file, load all file data to memory, then opens the same file again and writes new data. Imagine how much resources a script will need in case of a large file.

Node has another method to do this simpler - fs.appendFile.

fs.appendFile('log.txt', 'new logs', 'utf8', callback);

This is much better, but what does this method do? Let's discover how appendFile is implemented.
lib/internal/fs/promises.js:

async function appendFile(path, data, options) {
  // manipulations with the "options" argument, you can skip it
  // up to the return statement
  options = getOptions(options, { encoding: 'utf8', mode: 0o666, flag: 'a' });
  options = copyObject(options);
  options.flag = options.flag || 'a';
  return writeFile(path, data, options); // so, writeFile anyway?
}

// ...
async function writeFile(path, data, options) {
  options = getOptions(options, { encoding: 'utf8', mode: 0o666, flag: 'w' });
  const flag = options.flag || 'w';

  // in our case, the "path" isn't a FileHandle, it's a string
  if (path instanceof FileHandle)
    return writeFileHandle(path, data, options);

  // "fd" is a file descriptor (FileHandle instance)
  const fd = await open(path, flag, options.mode);
  return writeFileHandle(fd, data, options).finally(fd.close);
}

We discover what is the FileHandle a bit further.

Still, the appendFile does pretty much the same as we did earlier. In details, it:

• opens a file, gets a file handle
• writes data to a file (calls 'write' which decide whether to write buffer or string(C++ bindings)).

Is it okay to write logs like that? Not really. It's okay for occasional writes, here's why.

appendFile opens a file each time we need to write logs. In some cases, it can cause EMFILE error which means an operating system denies us to open more files/sockets. For example, if we need to write a new log entry every 5ms, a Node script will open a file every 5ms. Also, you need to wait for the callback to make appendFile again, otherwise, the function will append a file data in a conflicting way. Example:

// Notice: `appendFile` is called asynchronously
fs.appendFile('log.txt', '1', 'utf8', callback);
fs.appendFile('log.txt', '2', 'utf8', callback);
fs.appendFile('log.txt', '3', 'utf8', callback);

// log.txt can be as follows:
1
3
2

File descriptors

In short, file descriptor or file handle it's a reference to an opened file. They are non-negative integers. For example, standard input uses 0 value as a file handle, standard output uses 1, standard error output occupies 2 value. So, if you open a file programmatically, you'll get a file handle valued as 3 or more.
Node has its own wrapper for file handlers - FileHandle to perform basic operations on them (such as read, write, close, etc.).

The less opened file handles we have, the better. It means, fs.appendFile not a suitable solution to write logs.

Maybe streams?

Let's append to a file using writable streams:

// 'a' flag stands for 'append'
const log = fs.createWriteStream('log.txt', { flags: 'a' });

// on new log entry ->
log.write('new entry\n');

// you can skip closing the stream if you want it to be opened while
// a program runs, then file handle will be closed
log.end();

What did we do here? We create a writable stream that opens log.txt in the background and queues writes to the file when it's ready. Pros:

• we don't load the whole file into RAM;
• we don't create new file descriptors each time a program writes to the file. The purpose of streams here is to write small chunks of data to a file instead of loading the whole file into memory.

Summaries

• Don't use fs.appendFile if you need to write to a file often.
• Use fs.appendFile for occasional writes.
• Don't use fs.writeFile (or fs.write) to write a large amount of data or when dealing with large files. Use writable streams instead.

Source

Comments

[Adam DiCarlo]

Note this line in appendFile:

  options.flag = options.flag || 'a';

which opens the file in append mode, unless you passed a mode flag; it's not reading in (and rewriting) the whole file.

[Shivam Sharma]

How to handle file open/write errors in stream?

[Serhii]

Streams have on('error', handler) method to handle potential errors. E.g.

stream.on('error', (e) => console.log('I got an error!', e));

[Shivam Sharma]

Ok, thanks

[Louis Nel]

This was exactly what I needed for my current project! Thank you.