The latency of a Node.js service suddenly gets worse. p99 (the 99th percentile of response time: when 1000 requests are sorted from fastest to slowest, the value of the 990th) jumps. CPU looks low when you check overall usage. Memory is normal too. Nothing points to a cause. Endpoints that respond immediately, like a health check, are slow in the same way.
Node.js runs its event loop on a single thread. When one piece of work blocks, every other request waits in the queue. This congestion rarely shows up on an overall CPU graph.
I reproduced event loop starvation on GitHub Actions (ubuntu-24.04, 4 vCPU)1 and show, with the code I ran, which metric catches the degradation that an overall CPU graph tends to miss. In the same environment I also measured how much an overload guard (load shedding: returning 503 immediately once event loop delay exceeds a threshold) improves p99, and what it does not improve.
How Node.js handles requests
In an architecture that assigns a thread to each request, more concurrent connections mean more threads, and the cost of context switches (the overhead incurred each time the CPU switches from one thread to another) grows. Node.js avoids this by handling every request on an event loop that runs on one thread.
The event loop works through a to-do list (a queue) in order on a single thread. Each incoming request is added to the end of the queue, and the event loop takes items from the front and runs them. Work that has to wait for I/O (reading a file, network communication) leaves the queue while it waits and comes back when the I/O completes. In the meantime the loop can pick up the next item, so it can serve many connections without adding threads.
If one piece of work keeps hold of the thread, though, the rest of the queue has to wait until it finishes. This is event loop blocking. Reading a file with fs.readFileSync inside a request handler, running a shell command with execSync, and calling JSON.parse on a response several MB in size are all examples.2
How far this blocking backs up the queue is called lag. setTimeout(fn, 100) is a reservation to call fn after 100ms, but while the event loop is blocked, the call to fn is held back too. If fn should have been called after 100ms and is called after 450ms, the lag is 350ms.
Why overall CPU looks flat
A CPU graph shows usage for the whole system. Because Node.js runs on a single thread, a blocking operation that consumes CPU uses only one core's worth. On a server with 4 vCPU (virtual CPUs, the unit in which cloud environments divide physical cores and allocate them), one core at 100% shows up as only 25% overall.
I checked this by measurement, with two endpoints. /sync-cpu occupies one core with a busy loop (a loop that keeps hold of the CPU until its completion condition is met) that repeats JSON.parse until a deadline, blocking the event loop for a specified time. /light returns a response immediately with no computation. Ten concurrent connections sent a mix of requests, 1/3 to /sync-cpu and 2/3 to /light. The blocking time (sync_ms) was set to 50ms, 100ms, 150ms, and 200ms, and each setting was measured for 30 seconds. The columns to watch in the table are cpu_pct (CPU usage) and lag_p99 (p99 of event loop lag).
| sync_ms | /sync-cpu p99 | /light p99 | cpu_pct | lag_p99 |
|---|---|---|---|---|
| 0ms | 4ms | 4ms | 26.7% | 23ms |
| 50ms | 407ms | 353ms | 25.5% | 306ms |
| 100ms | 712ms | 1524ms | 25.9% | 612ms |
| 150ms | 1214ms | 1610ms | 25.6% | 764ms |
| 200ms | 1222ms | 2407ms | 25.7% | 1224ms |
cpu_pct stays almost fixed, between 25.5% and 26.7%. Quadrupling the blocking time from 50ms to 200ms barely moves CPU. lag_p99, on the other hand, is the event loop lag of the whole process, measured as a single value regardless of endpoint. The 23ms at 0ms is a floor that comes from the resolution: 20 setting; it is the baseline with no blocking. Lag rises with the blocking time: 306ms at 50ms, 612ms at 100ms, 1224ms at 200ms. CPU does not change, and only lag reflects the blocking.
At 200ms, the p99 of /light (2407ms) is higher than that of /sync-cpu (1222ms). /light has no blocking work of its own, but it sometimes waits a long time in the queue, and the larger the blocking time, the wider the spread of its p99.
Even when the overall CPU graph is flat, requests keep waiting in the queue if the event loop is congested. Looking only at overall CPU misses the congestion.
Spillover to unrelated endpoints
A Node.js process shares one event loop across all of its endpoints. While /sync-cpu is blocking the event loop, requests to /light wait in the queue too.
At 0ms, the p99 of /light is 4ms. At 50ms it jumps to 353ms. Nothing about /light itself changed. /sync-cpu occupied the event loop, and as a result the unrelated /light was kept waiting in the queue as well. The longer the blocking time, the worse the latency of /light gets.
Observing lag
Measuring event loop lag
Since Node.js v11.10.0, monitorEventLoopDelay() in node:perf_hooks gives you event loop lag.3
With resolution: 20, Node.js sets a timer that fires every 20ms, and each time it fires, it records the actual interval since the previous firing.4 If the event loop is blocked by other work, this timer cannot get in either. With a 300ms block, the timer does not fire during the block, and when it next fires, a single interval of a little over 300ms is recorded. With no blocking, intervals of around 20ms are recorded. These accumulate in a histogram (the distribution of how often each recorded interval occurs), and percentile(99) extracts the p99.
const { monitorEventLoopDelay } = require('node:perf_hooks');
const h = monitorEventLoopDelay({ resolution: 20 });
h.enable();
setInterval(() => {
console.log('lag p99:', h.percentile(99) / 1e6, 'ms');
h.reset();
}, 5000);
h.percentile(99) returns the p99 of the accumulated samples in nanoseconds, and / 1e6 converts it to milliseconds. h.reset() clears the samples and starts measurement for the next 5 seconds. Export this p99 as a metric and you can catch abnormal lag even when the CPU graph is flat.5
Per-core CPU
The blocking work in this experiment is a busy loop that repeats JSON.parse, and it keeps one core fully used. Overall CPU looks like around 25%, but a per-thread view with top -H or htop shows the Node.js main thread pinned at 100%, which confirms why the overall figure looked low.
When you run into the same symptom (low overall CPU, high p99), per-core CPU shows which thread the offending work is saturating.
Effect and limits of an overload guard
I implemented middleware that uses the measured lag to detect overload and return 503 (Service Unavailable, the HTTP status code that indicates the server temporarily cannot handle the request).
At startup it repeatedly sets a setTimeout with a 100ms interval. Each time the callback runs, it writes the difference between the planned 100ms and the actual elapsed ms into lagMs. If the event loop is blocked, this timer cannot get in either, so lagMs rises.
monitorEventLoopDelay can run at a 20ms interval without trouble because Node.js starts a libuv timer from the C++ side and does not call into JS for each measurement.6 A hand-written setTimeout loop, by contrast, creates a timer and runs a callback on the JS thread every time. The shorter the interval, the more of that overhead lands on the event loop itself, so the middleware's measurement uses 100ms.
let lagMs = 0
;(function measure() {
const start = Date.now()
setTimeout(() => {
lagMs = Math.max(0, Date.now() - start - 100)
measure()
}, 100).unref()
})()
On each request, the middleware compares lagMs with the threshold. If lag exceeds the threshold (150ms), it returns 503 immediately with Retry-After: 1. If lag is at or below the threshold, it passes the request to the next handler.
if (lagMs > thresholdMs) {
res.setHeader('Retry-After', '1')
return res.status(503).json({ error: 'overloaded', lag_ms: lagMs })
}
next()
lagMs is a single global variable in the process. /sync-cpu and /light both read the same value. Even when /sync-cpu is the one pushing lag up, the request that gets the 503 once the threshold is exceeded is the next one to arrive: whichever request enters the queue first, regardless of endpoint.
I applied the same load with this guard in place (threshold 150ms).
| sync_ms | /sync-cpu p99 | /sync-cpu 503 rate (%) | /light p99 | /light 503 rate (%) | lag_p99 |
|---|---|---|---|---|---|
| 0ms | 3ms | 0.0% | 3ms | 0.0% | 21.7ms |
| 50ms | 208ms | 95.6% | 203ms | 95.4% | 305.7ms |
| 100ms | 212ms | 98.7% | 202ms | 98.7% | 713.0ms |
| 150ms | 307ms | 98.9% | 162ms | 99.0% | 1072.2ms |
| 200ms | 401ms | 99.0% | 217ms | 99.0% | 1653.6ms |
p99 here is computed over all requests, 503s included. Since most of them are immediate 503s, this p99 is not a measure of how slow the requests that got through were. At sync=100ms, the /sync-cpu p99 goes from 712ms to 212ms, and the /light p99 from 1524ms to 202ms.
Look at the 503 rate. With a 150ms threshold, at sync=100ms, 98.7% of requests to both endpoints get a 503. Of the 18793 requests that reached /sync-cpu, only 239 got through. As long as the blocking work continues, lag piles up far beyond the threshold, and the service stays stuck rejecting most requests. The overload guard alone does not make a working service.
With the guard in place, lag_p99 did not go down, and at sync_ms of 100ms or more it went up.
| sync_ms | lag_p99 (no guard) | lag_p99 (with guard, threshold 150ms) |
|---|---|---|
| 50ms | 305.9ms | 305.7ms |
| 100ms | 612.4ms | 713.0ms |
| 150ms | 763.9ms | 1072.2ms |
| 200ms | 1223.7ms | 1653.6ms |
The guard turns new requests away with 503. It cannot stop blocking work that is already running on the event loop. The time /sync-cpu spends blocking the event loop continues regardless of the guard. The middleware's check and the setTimeout callback for measuring lag also run on the event loop, so while /sync-cpu is blocking, they cannot get in either.
Once lag exceeds the threshold, the guard returns 503 to every request regardless of endpoint. Even when /sync-cpu is what pushes lag up, /light gets rejected the same way.
The guard improves what requests that get through experience, but it does not resolve event loop starvation. Lag staying high shows this.
Finding and fixing the blocking work
Wherever the 503 threshold is set, lag does not go down as long as the blocking work keeps running. The overload guard limits inflow, and it leaves the work that congests the event loop untouched.
Finding it
The fundamental fix is to identify the work that is blocking the event loop and fix it. Once you have confirmed the symptom in production, bring it to a staging or development environment where you can reproduce the same load pattern. Start the process with node --prof, apply similar load, and after stopping it, process the sampling log with node --prof-process. This identifies the functions that occupy CPU time.7
Here is a profile taken in this test environment. It is the result of loading /sync-cpu?ms=200 with 10 concurrent connections for 20 seconds.
Statistical profiling result (27870 ticks, 2 unaccounted, 0 excluded).
[Bottom up (heavy) profile]:
ticks parent name
25652 92.0% node
25375 98.9% node
23368 92.1% JS: *blockEventLoop routes/sync-cpu.js:10
23368 100.0% JS: ^<anonymous> routes/sync-cpu.js:17
84% of all CPU ticks (the number of times the profiler, sampling at a fixed interval, observed where the CPU was executing) are concentrated in blockEventLoop. Following the callers also shows that it is called from an Express route handler. Confirming that blocking exists with lag, then narrowing down the responsible function with --prof-process, is the efficient order.
Fixing it
How to fix it depends on the kind of work. For synchronous I/O like fs.readFileSync or execSync, replacing it with the asynchronous fs.readFile or exec frees the event loop.
For work that uses CPU for a long time, the design of doing it inside a request handler on the application server needs to be reconsidered. Heavy work such as image processing or video conversion does not need to be synchronous with the response. The application server receives the request, puts a job on a queue, and immediately returns "accepted." The work itself lives outside the application server, in a separate worker process, Lambda, Cloud Functions, and so on. Because it is decoupled from the application server's event loop, the application server does not get congested however heavy the work is. For work that has to happen on the request path, such as JSON.parse on MB-sized data, moving it to a separate thread with worker_threads is an option. Before that, reconsider whether the application server should be handling that work at all.8
Running many Node.js application servers such as Express or Fastify as 1 vCPU containers, with a load balancer in front distributing requests, is a widely used setup. In this setup, when one container gets congested, the other containers keep accepting requests, so the single-thread constraint is absorbed operationally. The problem of one container's event loop getting congested does not go away, though. Requests assigned to that container stay queued inside it, and the same symptom shown in this article appears. Horizontal scaling solves the throughput problem; it does not resolve latency degradation inside one container.
Check whether the fix worked with lag_p99. Once the blocking work is removed, lag drops sharply.
Summary
The Node.js event loop handles every request on a single thread. When one piece of work blocks, every request in the process stops in the queue, and unrelated endpoints slow down uniformly.
An overall CPU graph does not show this congestion. On a 4 vCPU machine, one congested core shows up as only around 25% overall. Looking at CPU for this symptom will not find the cause.
What to check is lag. Export lag_p99 as a metric with monitorEventLoopDelay(), and you can detect blocking even when the CPU graph is flat. Looking at it alongside per-core CPU also shows which thread is congested.
An overload guard that returns 503 once lag exceeds a threshold improves response time, but lag itself does not go down, because the blocking work keeps running on the event loop. The guard buys time to find the root cause. Treat it as temporary protection until the blocking work is removed.
-
The results are here, and the code is in show-your-work. The Node.js version is 22.23.3. ↩
-
The core modules and JSON sections of the Node.js guide "Don't Block the Event Loop". ↩
-
The description of
monitorEventLoopDelay()in the Node.js 22 documentation. ↩ -
Histogram::RecordDelta()in the Node.js 22.23.3 source. It puts the difference from the previous recording time into the histogram as is. ↩ -
Each block produces only one sample, so a block that happens just once in a 5-second window does not show up in p99, only in
h.max. If heavy work runs only rarely, exporth.maxas well. A Node.js maintainer acknowledged this bias in nodejs/node#34661 and suggested reading the histogram together with Event Loop Utilization. The issue was closed with the bias left in place. ↩ -
CreateELDHistogram()andIntervalHistogram::OnStart()in the Node.js 22.23.3 source. The timer callback only callsRecordDelta()and does not enter JS. ↩ -
The profiling page of the Node.js guide. It also explains how to read the
[Bottom up (heavy) profile]section of the--prof-processoutput. ↩ -
The Offloading section of the Node.js guide "Don't Block the Event Loop", and
worker_threadsin the Node.js 22 documentation. ↩
Top comments (0)