loading...

Is true non-blocking code possible in practical Node development?

ankush981 profile image Ankush Thakur 惻1 min read

If you search on "top mistakes made by Node developers", "Blocking the Event Loop" is almost always the first one. And the example of code that works fine in other environments but is terrible in Node is given as (source):

[. . . ] a piece of CPU-bound code in a Node.js instance with thousands of clients connected is all it takes to block the event loop, making all the clients wait. CPU-bound codes include attempting to sort a large array, running an extremely long loop, and so on. For example:

function sortUsersByAge(users) {
    users.sort(function(a, b) {
        return a.age < b.age ? -1 : 1
    })
}

Now, this makes sense to me on a conceptual level, but I'm yet to come across any serious project where some computation is not involved. If not sorting, then agrresive filtering and merging are often required, and it's not always possible to do such tasks fully in the database.

So, my question to the Node developers is: If that's the case, how do you ever write truly performant code in Node? Or maybe these articles were written primarily for marketing and are blowing the problem out of proportion?

Posted on by:

ankush981 profile

Ankush Thakur

@ankush981

Fullstack dev working in Laravel, Node, Vue and React. Looking for bigger challenges!

Discussion

markdown guide
 

That's how we end up with AWS Lambda and such - the whole process just handles one client at a time. The async IO just queueing up data for that one request, not making other user's requests wait.
On a more local scale, that can take the form of using the node 'cluster' module to increase the chance some other thread is available to handle an incoming request while one is blocked.

Like always, profiling is important.
But so is the 99.99+ percentile response time.

To answer your question though

how do you ever write truly performant code in Node?

You don't. You aren't just in your one thread. (And not all computations scale.) You are in your one JavaScript thread, and there's no cure for that.

If you do have to calculate something, for isomorphic reasons for example, see if you can cache it.
You can consider worker threads, but think whether your network IO threads will just sit waiting for them.

 

Hello, thanks for the reply!

That's how we end up with AWS Lambda and such - the whole process just handles one client at a time.

Sort of what PHP has been doing all along? šŸ˜‹

If you do have to calculate something, for isomorphic reasons for example, see if you can cache it.

Which makes sense, but proper caching is a problem on its own.

To answer your question though

I guess my real question was: Is then Node a bad choice for building practical backends? And on a related note, are the async frameworks in Python, for example, repeating the same mistake? I mean, how is an async PHP/Python/Ruby server different from a Node server running on the cluster module?

 

Sort of what PHP has been doing all along? šŸ˜‹

cgi lmao

Is then Node a bad choice for building practical backends

There's lots of documentation and community. There are lots of premade integrations.
There's probably a module for doing exactly what you want to do.
It's pretty practical at the moment.

If you don't need any of that (or can afford not to make use of it and have special performance requirements) then there are plenty of alternatives.

You do seem to have strange expectations for the "backend" though.

What will this service be doing?
What kind of response times are we talking about?

What will this service be doing?

No, no, nothing specific. I was just wondering how to keep a single-threaded system from choking. And you've given me my answer: caching. šŸ˜ŠšŸ˜Š

  1. farming work out to long-running services while you just await and serve other requests. This is basically greenthreading
  2. horizontal scaling (clustering), emulates fully multi-threaded system, irrelevant whether on one machine or more
  3. caching

I say caching last because it's error-prone. It can be used as an emergency measure, but you should always keep an eye on where you're caching.
There are some things, like public pages, that it makes a lot of sense to cache, but that can even be done in a layer in front on nodejs, HTTP caching, not data layer caching. Of course, you may want to serve different random content on every reload, but then you just brought that on yourself :)

I say caching last because it's error-prone.

Why so? While I agree it's somewhat messy, data caching on the server-side seems like a pretty reasonable solution to me. šŸ¤” (I'm thinking of using something like Redis for storing computation results, and to avoid hitting the DB in particular.)

I do see that caching can become a problem as more and more data needs to be cached (eventually surpassing the available RAM), but even then, how bad can file-based caching be? šŸ¤”

  1. It's extra infrastructure (a form of complexity incidental to business logic)
  2. Cache invalidation (not serving stale data) is one of the famous "hard problems" of IT.
  3. In most cases it makes your system eventually consistent (Someone somewhere can get a stale response for a request that they initiated after the client modifying the data got a successful response.).
  4. Sudden onslaught of cache misses can cause cascading failure of your infrastructure. This can be due to anything from an emptied cache due to schema change to an extremely parallel request for a particular set of values, where the distributed cache passes millions of requests through before it has a chance to start cache a response for them.

Caches, and more specifically "Read Sides"/projections are inevitable in truly large systems, just as a form of eventual consistency is.
The problem is that developers often underestimate the complexity and concerns they introduce, and think they are just a no-brainer to "drop in" once there are are scaling issues.
Whereas in reality they can only safely be architected in with proper care.

 

TLDR: You can use generators.

non-blocking is used for file and networking IO. so for example in a non-threaded blocking sense a simple server looks like this:

// Pseudo code

var socket = listen_some_socket();
while (running) {
  // Here the application will wait (blocked) until a new connection made from client
  var client = socket.accept();

  // Here application will be blocked again until we read all the clients data over the socket
  var bytes = read_client_socket(client);
  var http_request = parse_http_request(bytes);
  var response = handle_request(http_request);

  client.write(response);
  client.close();
}

As you can see there is no callbacks / promisess etc. involved plain sequential code here. And if you look at the while loop, we can't accept any other requests until we respond to one client. For example client can be an attacker sending the request slowly on purpose, that means other people making request will be affected and wait for a long time. Or you may be running an expensive DB query, again other people trying to access your service will be blocked / waiting.

What nodejs or any other non-blocking languages does is using async io functions provided by the kernel, that means for example reading from a socket immediately returns if the socket is not ready and tells you to try again later. With an event loop like the nodejs has, you can put the current call stack on hold, and continue on other stacks (like handling other connections). This way you can program concurrently without dealing with threads. So instead of explicitly saying "hey js engine I can be suspended for a while, i'm waiting for IO anyways" IO operations in non-blocking languages tells this implicitly and js engine continues to work on other suspended routines.

So what happens when you run an infinite loop inside one of those routines. Like a simple for loop with some math calculations. math functions doesn't give a yield point to the engine so engine keep other routines on hold forever. The way you can tell the js engine to yield explicitly is using generators. You can move your expensive computation into a generator so you can yield after every iteration, or maybe after every 10 iterations. Now it is your responsibility to when to yield or when to continue computation.

 

Ah, yes, that's also a good point. I've come across generators in other languages/scenarios ... Thanks for the idea! :-)

 

I work for a NodeJS consulting company and meet many codebases/clients. I think this issue is a bit over exaggerated. Memory issues are far more often than blocking/CPU issues.

If you write stateless backend code (which you should) this is not really an issue. The devops team will put the correct machine types under your code and scale them horizontally depending on the load. In large prod Node apps you have multiple processes instead of multiple threads.

Writing stateful code is another story though. It will make horizontal scaling impossible since you have local pieces of data sitting in process instances which can not be shared with each other. It usually results in memory leaks sooner or later too. Moral of the story: keep the backend stateless and leave the rest to the devops team.

If you have periodically running expensive computations, having something dedicated is a good idea though. Workers and Lambdas are pretty nice in this case.

 

Pretty nice answer! But I'm a bit hazy on the stateful/stateless debate. Are sessions stateful (because they rely on the previous request) while token-based authentication is not? At least that's my understanding.

Which brings me to this:

Writing stateful code is another story though. It will make horizontal scaling impossible since you have local pieces of data sitting in process instances which can not be shared with each other.

I'm not sure I understand why this is a big deal. I can always store sessions outside of the server in, say, Redis cluster, and it will solve all problems (or at least it looks like that). Or maybe I've misunderstood "local pieces of data sitting on process instances". šŸ¤”

Please feel free to elaborate! :-)

 

I can always store sessions outside of the server in, say, Redis cluster, and it will solve all problems.

This is the key, storing state is obviously fine but you should never store state locally (in process memory). Sessions are inherently stateful, it doesn't matter if it is done with tokens or cookies. The key is that the state is not stored in the process memory but somewhere else (in a DB or client browser LocalStorage as examples.)

If you stick to this you will be able to scale your processes horizontally when it reaches the limit of its allocated resources. (In the case of this thread the resource is CPU I guess.)

Does this make sense?

I guess since I'm from the PHP world, it never occurred to me that people might try to use the process memory to store something! šŸ˜šŸ˜

Anyway, so that aside, if we build a Node app sensibly and there's some looping, averaging, etc., going on, it doesn't make a practical difference for small to medium projects, right? That's all I was aiming for through this post.