Introduction
Recently I worked to add a Typescript scaffold to the fastify-cli and I noticed that it provide, by using its 'start' command, the hot-reload for your application.
I kept looking at how the watcher works and found out that it uses a library which is, essentially, a wrapper around the 'watch' function of the Node module 'fs'.
I started thinking about it and, looking into the official Node doc, I started diving deeper and deeper trying to figure out how watchers work in Node.
After a few days here I am to let you implement a very simple hot-reload module and trying to explain what I understood about the 'watch' function of Node and how it is even possible for Javascript code to know when a file is changed.
But, before that, let me just introduce what hot-reload is for those of you who are not familiar with the term.
What is hot-reload?
When working on an application it is possible to make that application restart or reload every time we edit the code.
Let me explain this with an example:
// app.js
console.log('Hello world!')
To execute this file all I need to do is to run this script in my command line:
$ node app.js
// output
Hello world!
Now, if I want to change the message, for example, and print it again all I need to do is to run the script again.
// app.js
console.log('Hello universe!')
$ node app.js
// output
Hello universe!
Wouldn't be great if I could have something watching at my 'index' file and re-launching the script for me as soon as the code is changed?
Well, this is what hot-reload means.
Implementation
Let's take the first example's file as our target:
// app.js
console.log('Hello world!')
Node provides a really useful function to watch file changes in its 'fs' module which is called 'watch'.
This function takes a filename as first parameter and return an object called 'FSWatcher'.
FSWatcher extends EventEmitter class it will emit some events we can listen.
Note
If you don't know how EventEmitter class works in Node you can take a look at the official doc or you could consider this post I published.
Here is how our 'watcher' looks like:
// watcher.js
const fs = require('fs')
const watcher = fs.watch('app.js')
fs.on('change', () => console.log('changed'))
Running it:
$ node watcher.js
You will notice that the process doesn't stop until you stop it. This is because, off course, the 'watch' function keeps watching at the file until we say it to stop or we kill the process.
Now, while watcher is running, just try to edit the message in your 'app.js' file and look that 'changed' happens in your terminal.
Note
Seeing the 'changed' message appear twice for every change might be related on your system.
So now we have a 'watcher' which tells us when our application in modified.
Not so useful honestly, it would be better if it would reload our application immediately.
What we want is this:
$ node watcher.js
// output
Hello world!
// changing the message to 'Hello universe!'
//output
Hello universe!
One possible way to achieve this goal is to use the 'child_process' module provided by Node (for doc click here).
Let's start with the code:
const fs = require('fs')
const child = require('child_process')
// watch the target file
const watcher = fs.watch('app.js')
// create a child process for the target application
let currentChild = child.fork('app.js')
watcher.on('change', () => {
// we assure we have only one child process at time
if (currentChild) {
currentChild.kill()
}
// reset the child process
currentChild = child.fork('app.js')
})
So, what is happening here is:
- we spawn a new process which is child of the current process we are in using fork ('currentChild')
- every time we receive a 'change' event we kill the 'currentChild' and create another fork process
Consider this is not the only way to achieve the goal and, probably, not the best, but it is, in my opinion, the easiest to understand.
For a more in depth explanation of how 'process' module works please refer to the official doc.
Now, by executing our watcher in a shell:
$ node watcher.js
// output
Hello world!
// after editing the message inside app.js
Hello universe!
Under the hood
Now we have our simple implementation of an hot-reload module.
But how is it possible for the 'watch' function to know when the file changes?
First thing you may think is polling. You may think that Node takes somehow a screenshot of the current state of a file and, after some milliseconds, it compares with the current state of the file.
Well, this is what happens if you decide to use the watchFile function of 'fs' module.
But it's not our case.
Reading the documentation about the 'watch' function you'll encounter a paragraph titled Availability.
In this paragraph you can find that the operating system is able to notify filesystem changes.
This notifications, off course, are different basing on the OS we are using.
What I will try to do from now on is to follow this notifications (events) emitted by the operating system until we reach the 'JS land' where we can listen to them using Node's functions.
inotify
Let's consider we are using Linux for example.
As you can see, reading the Node official documentation, when Linux is the OS filesystem events are notified by inotify. The API provided by inotify provides a mechanism to monitor both files and directories. It can emit several events depending on what is happening:
IN_ACCESS
IN_ATTRIB
IN_CLOSE_WRITE
IN_CLOSE_NOWRITE
IN_CREATE
IN_DELETE
IN_DELETE_SELF
IN_MODIFY
IN_MOVE_SELF
IN_MOVED_FROM
IN_MOVED_TO
IN_OPEN
Since we are trying to watch when a file is edited we may consider the IN_MODIFY event.
If we keep reading until the 'Examples' section we can have the confirm which we took the correct event:
write(fd, buf, count);
Generates IN_MODIFY events for both dir and dir/myfile.
So essentially the IN_MODIFY event is generated when the file is written.
But we are still far away from the moment when we can use our JS code to listen to the event, so let's keep going with the next step.
If you are familiar with Node you should have heard of libuv.
libuv is the library Node uses for a lot of things, one of those things is filesystem.
Next step is looking for the event IN_MODIFY emitted by inotify inside libuv's source code.
libuv
Going into the GitHub repository of libuv and searching for IN_MODIFY will produce a single result in a file located at src/unix/linux-inotify.c, sounds familiar?
Yep, correct, this is the part of libuv which is responsible for the interaction with inotify and, in facf, scrolling down the file we can find this lines:
if (e->mask & (IN_ATTRIB|IN_MODIFY))
events |= UV_CHANGE;
if (e->mask & ~(IN_ATTRIB|IN_MODIFY))
events |= UV_RENAME;
It looks like our event IN_MODIFY is mapped to UV_CHANGE or UV_RENAME in libuv.
This sounds reasonable, so libuv maps different events coming from the OS (with different names) to the same events, this way Node and any other system using libuv will have to look for UV_CHANGE without considering the system it is running on.
What we finally have to do is to look for these 2 events in Node finally.
Node... finally
Events related to the filesystem are controlled by a module named FSEvent, this module can monitor a given path for changes and emit events basing on what happened:
- UV_CHANGE
- UV_RENAME
So, these 2 events are emitted by libuv when a certain path/file is changed or renamed.
The only thing we still need to see know is where Node takes these events to emit Javascript events we can listen to.
To answer this question what we can do is simply go into the Node source code.
So, let's go to the Node repository (here) and just use the GitHub search to look for UV_CHANGE inside the repository.
The first result of our search will bring us to a file called fs_event_wrap and, scrolling down this file, we will find what we were looking for in this comment where, basically, it is explained that libuv can set both UV_CHANGE and UV_RENAME at one time but the Node API allows only one event at time to be passed to the "JS land" so, if a UV_RENAME occurs, the UV_CHANGE will be ignored basically.
Finally we found where the libuv filesystem events handler is wrapped in Node.
But still no Javascript.
After a little more searching we can find lib/internal/fs/watchers.js file (a JS file) where we can notice a significant require instruction:
const { FSEvent } = internalBinding('fs_event_wrap');
Our answer might be in that 'internalBinding' function.
This function is in a file located at lib/internal/bootstrap/loaders.js.
This file (as explained in the comment at the beginning of the file itself) is compiled before the actual Node bootstrap and it is used to create, among other stuff, 'internalBinding' which is the private internal loader of C++ code.
So, the strange require instruction where 'internalBinding' is used should be now clear.
We can think 'internalBinding' as a 'require' for the C++ code in Node.
Going back to our lib/internal/fs/watchers.js and following the 'FSEvent' we reach line:
this._handle = new FSEvent();
and finally:
this.emit('change', eventType, filename);
which is exactly what we were looking for.
Conclusion
I hope you enjoyed my explanation but, please, consider I am note a professional Node developer and my goal is not to provide a production-ready implementation of a watcher off course.
My goal is just to, possibly, tease your curiosity, like mine have been writing this post, and suggest you to go deeper into things you think you are not fully understanding.
Please let me know if you think there are things that could be explained better (there should be) and, if you want, comment below with questions too and I'll do my best to answer.
Top comments (1)
Good explanatnion. Thnax.