Handling IO — NodeJS Event Loop Part 4

Welcome back to NodeJS Event loop series. In this post, I’m going to talk about how I/O is handled in NodeJS in detail. And I hope to dig deep into the implementation of the event loop and how I/O work in conjunction with other async operations as well. If you miss any of the previous articles in this series, I highly recommend you to go through them which I’ve listed in the following Roadmap section. I have described many other concepts in NodeJS event loop in the previous 3 posts.

Post Series Roadmap

• Event Loop and the Big Picture
• Timers, Immediates and Next Ticks
• Promises, Next-Ticks, and Immediates
• Handling I/O (This article)
• Event Loop Best Practices
• New changes to timers and microtasks in Node v11

Async I/O…. Coz blocking is too mainstream!

We are talking a lot about asynchronous I/O when it comes to NodeJS. As we discussed in the first article of this series, I/O is never meant to be synchronized.

In all OS implementations, they provide event notification interfaces for asynchronous I/O (epoll in linux/kqueue in macOS/event ports in solaris/IOCP in Windows etc.). NodeJS leverages these platform level event notification systems in order to provide non-blocking, asynchronous I/O.

As we saw, NodeJS is a collection of utilities which eventually are aggregated into the high performant NodeJS framework. These utilities include,

• Chrome v8 engine  — For high-performance JavaScript evaluation
• Libuv — For Event Loop with Asynchronous I/O
• c-ares  — For DNS Operations
• other add-ons such as ( http-parser , crypto and zlib )

Diagram1: Layered architecture of NodeJS

In this article, we’ll talk about Libuv and how it provides asynchronous I/O to Node. Let’s look at the event loop diagram again.

Diagram2: The Event Loop in a nutshell

Let’s recap what we learned so far about the event loop:

• Event loop is started with executing handlers of all expired timers
• Then it will process any pending I/O operations, and will optionally wait for any pending I/O to complete.
• Then it will move on to consume setImmediate callbacks
• Finally, it will process any I/O close handlers.
• Between each phase, libuv needs to communicate the results of the phase to the higher layers of Node architecture (which means JavaScript). Each time this happens, any process.nextTick callbacks and other microtask callbacks will be executed.

Now, let’s try to understand how NodeJS performs I/O in its event loop.

What is I/O?

Generally, any work which involves external devices except the CPU is called I/O. The most common abstract I/O types are File Operations and TCP/UDP network operations.

Libuv and NodeJS I/O

JavaScript itself has no facility to perform asynchronous I/O operations. During the development of NodeJS, libuv was initially started to provide async I/O to Node, although currently, libuv resides as a standalone library which can even be used individually. Libuv’s role in NodeJS architecture is to abstract internal I/O complexities and provide a generalized interface to upper layers of Node, so that Node can perform platform-independent asynchronous I/O without worrying about what platform it is run on.

A kind notice!

I recommend you to read the previous articles of this series if you do not have a basic understanding of the event loop. I might omit certain details here for brevity because I’d like to focus more on I/O in this article

I might use some code snippets from libuv itself, and I’ll only use Unix-specific snippets and examples only to make things simpler. Windows-specific code might differ a bit, but there shouldn’t be much difference.

I would assume you can understand a small snippet of C code. No expertise needed, but a basic understanding of the flow would be adequate.

As we saw in the previous NodeJS architecture diagram, libuv resides in a lower layer of the layered architecture. Now let’s look at the relationship between the upper layers of NodeJS and the phases of libuv event loop.

Diagram3: Event loop and JavaScript

As we saw in diagram 2 (Event loop in a nutshell) previously, there were 4 distinguishable phases of the event loop. But, when it comes to libuv, there are 7 distinguishable phases. They are,

1. Timers — Expired timer and interval callbacks scheduled by setTimeout and setInterval will be invoked.
2. Pending I/O callbacks — Pending Callbacks of any completed/errored I/O operation to be executed here.
3. Idle handlers — Perform some libuv internal stuff.
4. Prepare Handlers — Perform some prep-work before polling for I/O.
5. I/O Poll — Optionally wait for any I/O to complete.
6. Check handlers — Perform some post-mortem work after polling for I/O. Usually, callbacks scheduled by setImmediate will be invoked here.
7. Close handlers — Execute close handlers of any closed I/O operations (closed socket connection etc.)

Now, if you remember the first article in this series, you may be wondering…

1. What are Check handlers? It was also not there in the event loop diagram.
2. What is I/O Polling? Why do we block for I/O after executing any completed I/O callbacks? Shouldn’t Node be non-blocking?

Let’s answer the above questions.

Check Handlers

When NodeJS is initialized, it sets all setImmediate callbacks to be registered as Check handlers in libuv. This essentially means that any callback you set using setImmediate will eventually land in Libuv check handles queue which is guaranteed to be executed after I/O operations during its event loop.

I/O Polling

Now, you may be wondering what I/O polling is. Although I merged I/O callbacks queue and I/O polling into a single phase in the event loop diagram (diagram1), I/O Polling happens after consuming the completed/errored I/O callbacks.

But, the most important fact in I/O Polling is, it’s optional. I/O poling will or will not happen due to certain situations. To understand this thoroughly, let’s have a look at how this is implemented in libuv.

Ouch! It may seem a bit eye-twisting for those who are not familiar with C. But let’s try to get a glimpse of it without worrying too much about it. The above code is a section of uv_run method of which resides in core.c file of libuv source. But most importantly, this is the Heart of the NodeJS event loop.

If you have a look at diagram 3 again, the above code will make more sense. Let’s try to read the code line by line now.

1. uv__loop_alive — Check whether there are any referenced handlers to be invoked, or any active operations pending
2. uv__update_time — This will send a system call to get the current time and update the loop time (This is used to identify expired timers).
3. uv__run_timers — Run all expired timers
4. uv__run_pending — Run all completed/errored I/O callbacks
5. uv__io_poll — Poll for I/O
6. uv__run_check — Run all check handlers (setImmediate callbacks will run here)
7. uv__run_closing_handles — Run all close handlers

At first, event loop checks whether the event loop is alive, this is checked by invoking uv__loop_alive function. This function is really simple.

uv__loop_alive function simply returns a boolean value. This value is true if:

• There are active handles to be invoked,
• There are active requests (active operations) pending
• There are any closing handlers to be invoked

Event loop will keep spinning as long as uv__loop_alive function returns true.

After running callbacks of all expired timers,uv__run_pending function will be invoked. This function will go through the completed I/O operations stored in pending_queue in libuv event. If the pending_queue is empty, this function will return 0. Otherwise, all callbacks in pending_queue will be executed, and the function will return 1.

Now let’s look at I/O Polling which is performed by invoking uv__io_poll function in libuv.

You should see that uv__io_poll function accepts a second timeout parameter which is calculated by uv_backend_timeout function. uv__io_poll uses the timeout to determine how long it should block for I/O. If the timeout value is zero, I/O polling will be skipped and the event loop with move onto check handlers (setImmediate) phase. What determines the value of the timeout is an interesting part. Based on the above code of uv_run, we can deduce the follows:

• If the event loop runs on UV_RUN_DEFAULT mode, timeout is calculated using uv_backend_timeout method.
• If the event loop runs on UV_RUN_ONCE and if uv_run_pending returns 0 (i.e, pending_queue is empty), timeout is calculated using uv_backend_timeout method.
• Otherwise, timeout is 0.

Let’s not try to worry about different modes of the event loop such as UV_RUN_DEFAULT and UV_RUN_ONCE at this point. But if you are really interested in knowing what they are, check them out here.

Let’s now have a peek at uv_backend_timeout method to understand how timeout is determined.

• If the loop’s stop_flag is set which determines the loop is about to exit, timeout is 0.
• If there are no active handles or active operations pending, there’s no point of waiting, therefore the timeout is 0.
• If there are pending idle handles to be executed, waiting for I/O should not be done. Therefore, the timeout is 0.
• If there are completed I/O handlers in pending_queue, waiting for I/O should not be done. Therefore the timeout is 0.
• If there are any close handlers pending to be executed, should not wait for I/O. Therefore, the timeout is 0.

If none of the above criteria is met, uv__next_timeout method is called to determine how long libuv should wait for I/O.

What uv__next_timeout does is, it will return the value of the closest timer’s value. And if there are no timers, it will return -1 indicating infinity.

Now you should have the answer to the question “ Why do we block for I/O after executing any completed I/O callbacks? Shouldn’t Node be non-blocking? ”……

The event loop will not be blocked if there are any pending tasks to be executed. If there are no pending tasks to be executed, it will only be blocked until the next timer goes off, which re-activates the loop.

I hope you are still following me !!! I know this might be too much detail for you. But to understand this clearly, it is necessary to have a clear idea of what’s happening underneath.

Now we know how long the loop should wait for any I/O to complete. This timeout value is then passed to uv__io_poll function. This function will watch for any incoming I/O operations until this timeout expires or system-specified maximum safe timeout reaches. After the timeout, event loop will again become active and move on to the “check handlers” phase.

I/O Polling happens differently on different OS platforms. In Linux, this is performed by epoll_wait kernel system calls, on macOS using kqueue. In Windows, it’s performed using GetQueuedCompletionStatus in IOCP(Input Output Completion Port). I wouldn’t dig deep into how I/O polling works because it’s really complex and deserves another series of posts (which I don’t think I would write).

Some words about Threadpool

So far, we didn’t talk about the thread pool in this articles. As we saw in the first article in this series, threadpool is mostly used to perform all File I/O operations, getaddrinfo and getnameinfo calls during DNS operations merely due to the complexities of File I/O in different platforms (for a solid idea of these complexities, please read this post). Since the size of thread pool is limited (default size is 4), multiple requests to file system operations can still be blocked until a thread becomes available to work. However, size of the thread pool can be increased up to 128 (at the time of this writing) using the environment variable UV_THREADPOOL_SIZE, to increase the performance of the application.

Still, this fixed-size thread pool has identified to be a bottleneck for NodeJS applications because, File I/O, getaddrinfo, getnameinfo are not the only operations carried out by the thread pool. Certain CPU intensive Crypto operations such as randomBytes, randomFill and pbkdf2 are also run on the libuv thread pool to prevent any adverse effects on the application’s performance but, by which also makes available threads an even scarce resource for I/O operations.

As of a previous libuv enhancement proposal, it was suggested to make thread pool scalable based on the load, but this proposal has eventually been withdrawn in order to replace it with a pluggable API for threading which might be introduced in the future.

Some parts of this article are inspired by the presentation done by Saúl Ibarra Corretgé at NodeConfEU 2016. If you’d like to learn more about libuv, I would highly recommend you watch it.

Wrap Up

In this post, I described how I/O is performed in NodeJS in detail, diving into libuv source code itself. I believe the non-blocking, event-driven model of NodeJS makes more sense to you now. If you have any questions, I’d really like to answer them. Therefore, please don’t hesitate to respond to this article. And if you really like this article, I’d love it if you can clap and encourage me to write more. Thanks.

References:

• Official Libuv Documentation http://docs.libuv.org/
• NodeJS Guides https://nodejs.org/en/docs/guides/event-loop-timers-and-nexttick/
• Libuv Github https://github.com/libuv

Background Image Courtesy: https://i.imgur.com/JCVqX0Vr.jpg

---