DEV Community: Caio Borghi

React Query - why does it matter?

Caio Borghi — Mon, 01 Sep 2025 17:10:06 +0000

It avoids useEffect hell and handles: request state management, caching, refetching, retrying, "suspending" and error treatment; out of the box.

It helps with Asynchronous State management.

useEffect hell

You probably don't need useEffect, specially for handling requests.

Code difference

bad

import { useState, useEffect } from 'react';

const UniverseList = () => {
  const [isLoading, setLoading] = useState(true);
  const [error, setError] = useState(null);
  const [universes, setUniverses] = useState([]);

  useEffect(() => {
    const controller = new AbortController();

    const loadUniverses = async () => {
      try {
        setLoading(true);
        setError(null);

        const response = await fetch('/api/universes', {
          signal: controller.signal
        });

        if (!response.ok) {
          throw new Error(`Error: ${response.status} - ${await response.text()}`);
        }

        const jsonResponse = await response.json();
        setUniverses(jsonResponse.data || []);
      } catch (err) {
        if (err.name !== 'AbortError') {
          setError(err.message || 'Failed to load universes');
        }
      } finally {
        setLoading(false);
      }
    };

    loadUniverses();

    return () => {
      controller.abort();
    };
  }, []);

  if (isLoading) return <div>Loading...</div>;
  if (error) return <div>Error: {error}</div>;

  return (
    <div>
      {universes.map(universe => (
        <div key={universe.id}>{universe.name}</div>
      ))}
    </div>
  );
};

export default UniverseList;

good

import { Suspense } from 'react';
import { useSuspenseQuery } from '@tanstack/react-query';
import { ErrorBoundary } from 'react-error-boundary';

const FIVE_MINUTES = 5 * 60 * 1_000;
const TEN_MINUTES  = 10 * 60 * 1_000;

const fetchUniverses = async () => {
  const response = await fetch('/api/universes');

  if (!response.ok) {
    throw new Error(`Error: ${response.status} - ${await response.text()}`);
  }

  const jsonResponse = await response.json();
  return jsonResponse.data || [];
};

const UniverseListContent = () => {
  const { data: universes } = useSuspenseQuery({
    queryKey: ['universes'],
    queryFn: fetchUniverses,
    staleTime: FIVE_MINUTES,
    gcTime: TEN_MINUTES,
    retry: 3,
    retryDelay: attemptIndex => Math.min(1000 * 2 ** attemptIndex, 30000),
  });

  return (
    <div>
      {universes.map(universe => (
        <div key={universe.id}>{universe.name}</div>
      ))}
    </div>
  );
};

const ErrorFallback = ({ error, resetErrorBoundary }) => (
  <div role="alert">
    <p>Something went wrong:</p>
    <pre>{error.message}</pre>
    <button onClick={resetErrorBoundary}>Try again</button>
  </div>
);

const UniverseListWithSuspense = () => {
  return (
    <ErrorBoundary
      FallbackComponent={ErrorFallback}
      onReset={() => window.location.reload()}
    >
      <Suspense fallback={<div>Loading universes...</div>}>
        <UniverseListContent />
      </Suspense>
    </ErrorBoundary>
  );
};

export default UniverseListWithSuspense;

Why bad?

Repeated code, required to manage async state, will spread as garden weeds as the project scales.

If you're not willing using React Query, at least create your own decoupled hooks and make sure to properly test them.

caching

React Query can cache your endpoints results and expire then after a configurable expiration time.

It also allows you to tie tags with queries and invalidate them on mutations.

refetching

As easy as calling a function, as it should be.

const UniverseListContent = () => {
  const { data: universes, refetch } = useSuspenseQuery({
    queryKey: ['universes'],
    queryFn: fetchUniverses,
    staleTime: FIVE_MINUTES,
    gcTime: TEN_MINUTES,
    retry: 
    retryDelay: attemptIndex => Math.min(1000 * 2 ** attemptIndex, 30000),
  });

  return (
    <div>
      <button onClick={refetch}>Reload All</button>
      {universes.map(universe => (
        <div key={universe.id}>{universe.name}</div>
      ))}
    </div>
  );
};

Go 1.24 uses Swiss Tables, what are they?

Caio Borghi — Thu, 20 Feb 2025 01:13:28 +0000

Introduction
The Old Map
- Chaining
  - Practical Example
  - Problem
Swiss Table
- Linear Probing
- Steroids (SSE3)
- Metadata
- Problem
Elastic Hashing
- What is it?
Conclusion
References

Introduction

In v1.24, Go replaced its Map implementation with a new version of hash table called Swiss Table, or flat_hash_map.

What is a Swiss Table?
A Swiss Table is a map, that uses a cache-friendly, more efficient (with shorter memory footprint) approach that makes comparisons and insertions faster.

It also uses a different strategy for addressing collisions: linear probing on steroids over the previous strategy chaining.

When working with hash-tables, one thing is certain: there will be conflict.

The Old Map

The previous implementation was highly tuned for memory efficiency and performance.

Go team is awesome.

Chaining

How does it work? It pre-allocates memory in the form of buckets, where each bucket can have up to 8 key-value pairs. When a bucket is full (or half-full), the algorithm allocates a new overflow bucket as a linked list, in a process known as chaining.

As the table approaches a high rate of load factor*, the runtime moves all entries to a new memory address block (usually twice as big as the previous one), this is known as rehashing.

* Load Factor = used positions / total capacity

In traditional chaining, whenever there is a conflict, the runtime allocates memory for a new node/key-value pair and stores it as a linked-list.

I'll focus on how chaining works for nodes and leave the buckets strategy explanation for the Go 1.23 Map Implementation.

Practical Example

Imagine a hash function that, based on a string, returns an index (offset value) to a fixed memory address:

x = len(x)

Usually, these functions try to produce an avalanche effect to distribute the results "evenly" over the map memory interval.

Our hashing function is more vulnerable to conflicts, as it only returns the size of the string to define the address.

Now, let's assume we add the following keys:

key	index
C	1
JS	2
PHP	3

And this table:

index	key	memory	next
0	nil	0	nil
1	C	1000	nil
2	JS	1008	nil
3	PHP	1016	nil
4	PERL	1024	nil

When we add a new "Go" key, we caused a conflict.

As the index of "Go" is 2 and there is already "JS" at position 2, the strategy will allocate a new node in the available memory and point the next prop of "JS" to "Go".

index	key	memory	next
0	nil	0	nil
1	C	1000	nil
2	JS	1008	2064
3	PHP	1016	nil
4	PERL	1024	nil
__	Go	2064	nil

So we have keys, that are used as inputs in a hash function, that generates an index (or an offset for a fixed memory address/start of the list) and the keys or nodes or buckets are inserted as linked-lists.

Problem

As you can see, each new node is placed far from it's conflict key in the memory.

This means that, when searching for a key that has conflicts (Go, for example), the processor will fetch sparse addresses in memory.

Even though the Go previous implementation used a lot of performance improvement techniques (as using 8-sized buckets instead of single-nodes) and partially comparing keys (7 bits instead of 64), this chaining approach is not cache-friendly.

You can check the specific details here.

Swiss Table

It's a cleaver implementation that uses 1 byte metadata and linear probing on steroids.

Linear Probing

No more dynamic memory allocation for buckets with linked lists.

Consider this table:

index	key	memory
0	nil	1000
1	C	1008
2	JS	1016
3	PHP	1024
4	PERL	1032
5	nil	1040
...	...	...
N	nil	N_ADDRESS

A better visualization in this case is to look at the has table as a, well, _flat_hash_map.

When a conflict occurs, the algorithm will linearly search for the next position, one by one.

For adding "Go", we have:

hash("Go") = 2
See slot 1016 and realize it is filled.
See slot 1024 and realize it its filled.
See slot 1032 and realize it its filled.
Slot 1040 is open, use it.

It's only on slot 1040 that will find an open spot, this is where the "Go" key goes.

The important thing to notice is that this time, collisions are located nearby each other, making them cache friendly, which speeds things up a little bit.

Steroids (SSE3)

Streaming SIMD Extensions 3 is a kind of processor instruction that, combined with curated software engineering techniques such as bit manipulation, provides a way to perform parallel reads of memory addresses with a single instruction.

Which means that, when probing for collisions, the Swiss Table could perform up to 16 checks at once, even though Golang seems to be using just 8, it's way better than checking one by one!

Metadata

Swiss Table uses a metadata byte to partially store the hashed key, enabling quick comparison (7 bit instead of 64).

With linear probing (also known as open addressing) + metadata chunking + SSE3, comparisons and inserts are faster and consume less memory than the previous implementation.

1.23.4 vs 1.24

I ran this article benchmark script on my laptop.

The test was:

Create and populate a 1_000_000 items map
Performs 10_000_000 lookups using mod indexing.
Inserts 1_000_000 new entries into the map.
Removes the first 1_000_000 from the map.

Results

Operation	Go 1.23 (ms)	Go 1.24 (ms)	Improvement (%)
Lookup	287.340375	184.787167	35.67% faster
Insertion	118.564333	66.4095	43.99% faster
Deletion	39.893875	61.364875	-53.85% slower

In Go 1.24’s Swiss Tables: The Silver Bullet or Just a Shiny New Gadget? article, we can also see the efficiency of Swiss Tables in Memory Usage.

Problem

Even though the Go Swiss Table uses buckets and a sparse hash function (for the avalanche effect) to distribute keys evenly over the pre-allocated memory block, as the table get's full, conflicts will generate clustered sections that will increase search time.

It's easy to see this happening with a 10 positions table:

Table: [nil, C, JS, PHP, PERL, nil, nil, nil, nil, nil]

Adding a new "B" key would cause:
- len(B) = 1
- position 1 is occupied by "C", look next
- position 2 is occupied by "JS", look next
- position 3 is occupied by "PHP", look next
- position 4 is occupied by "PRL", look next
- position 4 is free, add "B"

Table: [nil, C, JS, PHP, PERL, B, nil, nil, nil, nil]
Index:   0,  1,  2,  3,   4,   5,   6,   7,   8,   9

Now, any hash function call that returned 1, 2, 3, 4 or 5 would require searching up to 4 positions, piece of cake for the SSE3, I recognize, but what if there was a better way?

Elastic Hashing

In January 2025, a new paper regarding open addressing was launched suggesting a new approach, theoretically better than linear probing, called Elastic Hashing.

Feel free to read the news article or the paper to get the full context.

It claims to beat a 40-years old conjecture (Yao's conjecture), that defends linear probing as a simple, with near-optimal efficiency, that doesn't degrade catastrophically as load increases.

Krapivin discovered the new strategy while being unaware of Yao's conjecture, which indicates we should challenge known constraints more often.

What is it?

Basically, instead of checking positions one by one, or 16 by 16 (as our beloved Swiss Tables do), it created a new bidimensional strategy to calculate the insert address using virtual overflow buckets.

The idea is, there is a new function named φ(i, j) that returns a position of a node, where:

i = Primary Bucket (Hash Result)
j = Overflow Count

So, using our previous hash function, both keys "JS" and "Go" would return 2 as their "Primary Bucket".

The insertion order would determine if the key would be placed at position φ(2, 1) = 2 or maybe φ(2, 2) = 7.

Magic

The magic lies in the φ function, that is able to virtually create overflow buckets for collisions_. It outperforms the complexity algorithm for linear probing for worst and average cases, with these jumps or wormholes, it probes less addresses, leading to a better theoretical insert performance.

Will Elastic Hashing be applied to Swiss Tables?
Maybe. I hope so. Time will tell.

I still have open questions:

Will elastic hashing prove itself faster than linear probing + SSE3?
Can Elastic Hash benefit itself from the parallel reads of SIMD?
Which language will be the first to implement this new algorithm?

If you, by any chance, happens to cross some of these answers out there, leave it a comment!

Conclusion

It's nice to experience these latest improvements to Hash Table efficiency as they happen, in real time.

A one-month old paper improving the code complexity of one aspect of the core data structure algorithm is great news!

It shows that there are no boundaries for performance improvements, not even decade-old rules are safe, no matter where you are, there is always room for improvement.

Cheers.

References

Benchmarking DeepSeek R1 on a Developer’s MacBook

Caio Borghi — Tue, 04 Feb 2025 20:00:42 +0000

Introduction
- What's the point?
The Prompt
- What is Time?
The Answer
How were the metrics gathered?
- Sequence Diagram
- Process Monitor
- GPU Monitor
- Requests Metrics
- Hardware Specification
- Tools
Benchmark Results
- Waiting Time (TTFB)
- Velocity
  - Comparing different token/s speed
- Acceptable Thresholds
- Throughput + Wait Time
- Duration
- Combined Metrics (Tokens/s x Duration x Wait Time)
- GPU Usage
- RAM/CPU/Threads Usage
Summarized Results
- How many tokens/s can I get running DeepSeek R1 Gwen 7B locally with ollama?
- How many parallel requests can I serve with reasonable throughput?
- What is a reasonable throughput?
- How does the number of concurrent requests impact the performance?
Conclusion

Introduction

Hey there,

With the new hype of AI (Jan 2025) over DeepSeek's high-quality open-source models, an urge to explore self-hosted LLM models infected my mind.

Therefore, I decided to build a stress test benchmarking tool with Go (Channels 💙), fire it against Ollama & DeepSeek, monitor a bunch of metrics, and share the results with you.

This post analyzes the throughput capacity of DeepSeek-R1-Distill-Qwen-7B model running on Ollama, at my personal dev MacBook and M2 Pro with 16GB Ram and a 19-Core GPU.

Oh, the project is open source and can be found at ocodista/benchmark-deepseek-r1-7b on GitHub (drop a ⭐ if you think this type of content is useful 😁✌️).

What's the point?

I wanted to see how many parallel requests my M2 could handle at a decent velocity and experiment with Go + Cursor + Claude Sonnet 3.5.

It was a great experience and although the majority of the code was written by AI, none of the docs (or this text) was.

You can expect this experiment to answer the following questions:

How many tokens/s can I get running DeepSeek R1 Gwen 7B locally with ollama?
How many parallel requests can I serve with reasonable throughput?
- What is a reasonable throughput?
How does the number of concurrent requests impact the throughput?
How much power did my GPU used while running this study?

Now let's talk about the test.

The Prompt

Inspired by a recent book I've read (A Universe From Nothing), I selected the following question as the prompt for each request:

What is the philosophical definition of time?

Which is a very profound question that requires some reasoning. It is useful to analyze the Chain of Thought process of DeepSeek R1, as it is a core feature of the model to return the answer in 2-steps:

<think>{THINK}</think> and {RESPONSE}.

What is Time?

Time has abstract (the final end, the first beginning) and structured (seconds, minutes, hours) definitions.

It can be used to express a relation between unrelated events, to represent something we feel (the passage of time) when we access our memories, and to wonder about the big mysteries of the universe: Where do we come from? Where are we going?

I am the beginning, the end and the middle.
— Raul Seixas

The selected structured representation of time was min:seconds and we'll analyze Waiting Time and Duration Time.

The Answer

Here you can see one of the responses generated by the 7B model during one of the tests.

My opinion (as a Software Engineer, not a philosopher nor a physicist) is that is pretty good.

It's marvelous to read the <think> section and observe how the model groups multiple subjects related to the question before providing a final answer.

I'm not a Data Science expert, so I can't explain the under-the-hood workings, but it appears to reuse this first exploration prompt to re-prompt the model.

This is revolutionary, as it's the first time I've seen this two-step answer strategy built inside the model.

The strategy itself isn't necessarily new, as I've manually used it before with a Custom GPT called Prompt Optimizer, a kind of pre-prompting to get better final prompts, it is specially helpful when generating images from text.

Anyway, this is pretty cool!

The difference in quality between DeepSeek R1 (full model) and ChatGPT for small prompts is noticeable.

This automatic context-universe-expansion also eliminates the growing need to be good at Prompt Engineering. It now comes for free, inside the model.

So, returning to the test 😁

If you don't care about how data was collected, you can time-travel to the Benchmark Results and watch some good-looko charts.

How were the metrics gathered?

The idea was to execute multiple rounds of parallel HTTP requests to the Ollama Web Server endpoint (1, 2, 4, 8, 16, 19, 32, 38, 57, 64,76, 95, 128 and 256).

Since my GPU has 19 cores, I selected 19 as one of the rounds (and a few other multiples) to ensure each GPU Core is busy.

Sequence Diagram

The test goes in cycles, each cycle containing a different set of concurrent requests.

Each cycle waits for 10s after finishes to start the next one.

Process Monitor

It uses pgrep ollama to find all PIDs involved in running the model requests and will monitor, store and display:

Thread Count
File Descriptors
RAM Usage
CPU Usage

GPU Monitor

It uses the awesome tool powermetrics to calculate:

Power (W)
Frequency (MHz)
Usage (%)

Over 1s intervals.

Requests Metrics

For each request of the cycle, it was analyzed the following properties:

Throughput (Tokens/s)
TTFB (Time To First Byte)
WaitingTime
TokenCount
ResponseDuration
TotalDuration

Hardware Specification

Component	Specification
Device	MacBook Pro 16-inch (M2, 2023)
CPU	12-core ARM-Based Processor
Memory	16GB RAM
GPU	Integrated M2 Series GPU (19 Cores)
OS	MacOS Sonoma

Tools

Ollama

A local LLM inference framework CLI that's very easy to use.

For this benchmark, the selected model was deepseek-r1:7b but it could've been any other, as ollama makes it ridiculously simple to run LLMs locally.

Golang

The chosen programming language, used to create the benchmarking client and monitoring tools.

Why? Well, Go is an outstanding tool for parallel computing.

I'm a JS developer (not a Golang expert yet) but I can recognize a great parallel/concurrent tool when I see one.

Go Scheduler is indeed awesome.

Python

There is nothing better than Python to analyze a bunch of CSV files and generate beautiful charts.

For instructions on how to run this benchmark, please check how-to-run.md.

Benchmark Results

As we are using HTTP requests as the traffic method of this experiment, I decided to use Time To First Byte to represent the Waiting Time.

Waiting Time (TTFB)

The waiting time (TTFB) is the delay between hitting the Enter key and seeing the first character on your screen.

Since in this experiment the client and the server are on the same network, hardware and computer, we are actually calculating the time it takes for the Golang process to communicate with the Ollama process, that will be running the DeepSeek R1 model, that will generate the responses and stream it back to Golang process.
With that said, let's contemplate some colored charts:

Ok, looks interesting, it shows that somewhere near 25 parallel requests the Wait Time spikes aggressively and continues to grow at a stable rate, reaching an unbelievable 50 minutes on p99 for 256 requests/s.

Let's zoom in:

We can see that from cycle 1-19, the TTFB is close to 0, ranging its average from 0.62s to 0.83s, that's pretty much instant for human perception.

It makes sense if you take the available amount of GPU cores into consideration, which you should, otherwise the OLLAMA_MAX_PARALLEL flag will have it's default value (4), and your results will be poisoned (trust me, I've been there).

TTFB Table Data

TTFB Table Data (only if you care)

Parallel	Avg	P95	P99
1	0.83	0.93	0.93
2	0.36	0.44	0.44
4	0.38	0.42	0.42
8	0.47	0.51	0.51
16	0.64	0.67	0.67
19	0.62	0.64	0.64
32	84.05	227.13	234.90
64	321.96	761.89	790.82
128	858.27	1839.99	1915.72
256	1247.02	3078.62	3211.85

Velocity

This is the most important metric of the experiment.

The following chart shows that by running DeepSeek R1 Gwen 7b, self-hosted with Ollama on a MacBook Pro M2 with 16GB RAM and 19 GPU cores, we can achieve a maximum of 55 tokens/s when making a single request.

When utilizing the full potential of the GPU with 19 parallel requests though, the average throughput dropped to mere 9.1 tokens/s.

It kept dropping until the lowest value at 256 requests with 6.3 tokens/s.

Comparing different token/s speed

It's hard to mentally visualize what 55, 9.1 or 6.3 tokens/s actually means, so I recorded a couple of GIFs to help:

55 tokens per second

30.7 tokens per second

9.1 tokens per second

For me, the ideal speed of a fast application would be around 100 tokens/s, and the slow-but-usable floor limit would be around 20 tokens/s. I mean, faster speed is never enough. It's like internet download/upload or video FPS (Frames Per Second) when rendering games, the higher the better.

100 tokens/s

Acceptable Thresholds

What should be the acceptable thresholds for an usable real-world application using the DeepSeek + Ollama?

How much time does an average user waits on a loading application before it quits?

What is the slowest acceptable speed to read a text without getting bored?

Maximum Acceptable Wait Time

I'll choose 10s as an arbitrary value for the maximum acceptable wait time.

In reality, users are more impatient and the value may be much lower.

Looking at the TTFB table data, 19 is the last cycle that meets our 10s threshold with a 0.62s wait. At 32 parallel requests, the average Waiting Time jumps to 1 minute and 25 seconds.

Unless you're using DeepSeek for background tasks with no human interaction, a wait time of 1m25s is unacceptable. Based on the 10s threshold, the maximum parallel requests for a usable app should be 19.

Minimum Acceptable Response Speed

I believe it should be ~19.9 tokens/s.

This metric is totally arbitrary and personally chosen based on how I felt watching the speed gifs.

Anything less than 20 tokens/s feels slightly annoying.

With this new limit, the maximum parallel requests considering acceptable response time is 4.

Ok, let's take a look at the combined metrics now.

Throughput + Wait Time

Speed is good, but in 2025, Time To First Byte must be minimal in order for a product to be usable.

No one likes to click a button and wait 20 seconds or 2 minutes for something to happen.

Throughput + Wait Time Table Data

Table Data

Parallel	Avg t/s	P95 t/s	P99 t/s	Wait(s)	Duration	P99 Duration
1.0	53.1	53.1	53.1	0.94	00:39.33	00:39.33
2.0	30.7	30.9	30.9	0.36	01:12.09	01:12.27
4.0	19.9	20.4	20.5	1.54	01:34.31	01:47.17
5.0	19.2	20.6	20.7	0.30	01:39.01	01:59.26
8.0	12.9	13.5	13.7	0.47	02:32.12	02:54.62
16.0	9.8	10.1	10.3	0.64	03:26.81	04:41.07
19.0	9.1	9.5	9.6	0.62	03:43.55	05:23.83
32.0	8.0	9.3	9.3	84.05	04:08.64	06:15.66
64.0	7.1	9.0	9.3	321.96	04:39.51	07:30.13
128.0	6.5	8.9	9.2	858.27	05:06.60	07:28.05
256.0	6.3	8.8	9.4	1534.79	04:12.89	07:52.97

Duration

The first cycle lasted less than a minute (39 seconds) while the last cycle took 8 min and 51 seconds to complete.

Requests Average Duration

For the initial cycles, the average duration of each request grows slowly but noticeably. While the single request took only 39s to complete, requests that used all available cores (cycle 19) took, on average, 03 minutes and 43 seconds to complete.

Table Data

Parallel	Min	Avg	P99	Max
1	00:39.33	00:39.33	00:39.33	00:39.33
2	01:11.92	01:12.09	01:12.27	01:12.27
4	01:19.14	01:34.31	01:47.17	01:47.17
5	01:24.99	01:39.01	01:59.26	01:59.26
8	02:06.97	02:32.12	02:54.62	02:54.62
16	02:01.12	03:26.81	04:41.07	04:41.07
19	02:14.66	03:43.55	05:23.83	05:23.83
32	02:41.05	04:08.64	06:15.66	06:15.66
64	02:17.05	04:39.51	07:30.13	07:30.13
128	02:35.36	05:06.60	07:28.05	08:38.04
256	00:00.00	04:12.89	07:52.97	08:51.75

Combined Metrics (Tokens/s x Duration x Wait Time)

Wait time grows linearly after 19 parallel requests, making it unusable for interactive applications.

It also shows that throughput drops exponentially before stabilizing at much lower cycles.

Let's zoom in into smaller cycles:

Here, the chart is different: the wait time is stable at <1s until cycle 19 and it's clear to see the connection between throughput and p99 request duration.

GPU Usage

Thanks to powermetrics, it's possible to get GPU usage metrics in MacOS!

The Macbook M2 19-GPU proved to be pretty constant, with small variations in the GPU Frequency, stable at 1397MHz, and Power Usage, stable at ~20.3W.

The concurrency level didn't seem to affect the GPU metrics.

sudo powermetrics --samplers gpu_power -n1 -i1000

RAM/CPU/Threads Usage

To analyze how much computer resource the Ollama + DeepSeek was consuming, I tracked the ollama processes (with pgrep, lsof and ps) and monitored the following metrics:

CPU Usage (%)
Memory Usage (%)
Resident Memory (MB)
Thread Count (int)
File Descriptors (int)
Virtual Memory Size (MB)

Ok, when you have ollama serve running idle, it uses a single process.

When there is 1 or 256 active requests, ollama uses 2 processes.

By analyzing the chart, we can see that they behave different from each other.

While one of them has high memory/cpu usage and low thread count/open file descriptors, the other one has the opposite: low cpu/memory usage with linear growing open FDs and high thread count.

If I had to guess, I would say that the green process may be responsible for the Web Server while the red one for the DeepSeek R1 LLM generations.

Web Server Process

While the open File Descriptors grow linearly as the number of concurrent requests grows, the Thread count has a steeper pattern upward.

Notably, the CPU and memory usage of this process remain constantly low, between 0.2~0.6% for CPU and 82.6 -> 114.1MB for RAM.

Table Data

Concurrency	Avg CPU%	Max CPU%	Avg Mem%	Max Mem%	Avg Threads	Avg FDs	Avg RAM(MB)	Max RAM(MB)
1	0.6	22.8	0.2	0.5	17.0	18.9	36.4	82.6
2	0.0	0.8	0.5	0.5	18.0	21.0	83.4	83.6
8	0.2	1.7	0.6	0.6	20.0	31.7	90.4	90.9
16	0.2	1.5	0.5	0.6	21.0	42.5	79.4	95.2
19	0.2	2.2	0.5	0.5	21.0	44.9	82.0	82.6
32	0.2	2.2	0.5	0.5	21.0	48.9	85.6	86.2
64	0.2	2.9	0.6	0.6	21.0	69.1	92.1	94.6
128	0.2	2.7	0.6	0.7	37.0	102.3	103.8	108.4
256	0.2	4.2	0.7	0.7	72.0	145.4	108.2	114.1

DeepSeek Process

If the Open FDs and Threads Count snitched the Web Server process, the Memory Consumption and maximum Threads Count of 18 snitched the DeepSeek Process.

The fact that the number of threads doesn't outgrow the number of GPU Cores, even when under higher concurrency cycle, indicates that this process might be the one in charge of DeepSeek, that uses the GPU, that has 19 cores.

The average RAM Usage of this process is remarkable: from 2.2 to 2.3GB, representing 13.7 to 14.6% of all available RAM, CPU usage is also high for a process, consuming 5.7% for a single request and 13.1% for 256.

Table Data

Concurrency	Avg CPU%	Max CPU%	Avg Mem%	Max Mem%	Avg Threads	Avg FDs	Avg RAM(MB)	Max RAM(MB)
1	4.6	5.7	13.7	13.7	12.1	22.0	2248.1	2250.6
2	4.2	5.2	13.8	13.8	16.0	23.0	2259.9	2263.1
8	4.9	7.9	14.0	14.1	16.0	28.5	2299.6	2303.3
16	5.7	11.6	14.0	14.3	16.4	34.1	2299.1	2335.5
19	5.8	14.3	14.2	14.2	17.0	35.3	2327.4	2330.6
32	5.0	12.0	14.4	14.4	17.0	34.8	2359.1	2366.3
64	5.5	14.2	14.5	14.6	17.1	37.7	2374.2	2390.0
128	5.5	13.2	14.4	14.6	18.0	38.8	2366.5	2397.7
256	5.5	13.1	14.4	14.6	18.0	39.5	2364.1	2385.5

Summarized Results

These results were generated running Ollama + DeepSeek on a Macbook M2 Pro, 16GB of RAM and 19-Core GPU, it will probably be different in a different setup.

How many tokens/s can I get running DeepSeek R1 Gwen 7B locally with ollama?

For a single request: 53.1 tokens/s.

For 19 parallel requests -> 9.1 tokens/s.

For 256 concurrent requests -> 6.3 tokens/s.

You can check the Table Data if you want.

How many parallel requests can I serve with reasonable throughput?

Assuming 19.9 tokens/s as a reasonable throughput, this machine can serve up to 4 requests in parallel.

This may be enough for single-person daily routine tasks but is definitely not enough to run a commercial API server.

What is a reasonable throughput?

During the course of writting this article, I created a CLI to showcase different token/s speeds, you can check it out here.

How does the number of concurrent requests impact the performance?

A lot!

For more than 19 concurrent requests, the wait time becomes unbearable, and for more than 5 parallel requests, the response speed is too low.

Table Data

Conclusion

Considering this is the 7b version model, and it can reach up to 55 tokens/s when serving a single request, I would say it is fast and good enough to chat interactively during daily tasks, with a reasonably low power usage (same as a led lamp).

I mean, the quality is far from great when compared to 671b version (which is the model that beats OpenAI models), but I believe this is just the beginning.

Quantization strategies will become more effective, and soon we'll be able to cherry-pick subjects to train smaller models, which we demonstrated to be possible to run on a developer's computer. It is possible, we're in the Tech Industry.

It happened with the processor, the disk and the memory, it's a matter of time until it happens with AI inference chips and LLM models.

That's it for today, thank you for the reading 😁✌️!

The Evolution of React State Management: From Local to Async

Caio Borghi — Tue, 20 Aug 2024 20:37:43 +0000

Introduction
Local State
- Class Components
- Functional Components
- useReducer Hook
Global State
- What is Global State?
- How to Use It?
- The Main Way
- The Simple Way
- The Wrong Way
Async State
Conclusion

Introduction

Hi!

This article presents an overview of how State was managed in React Applications thousands of years ago when Class Components dominated the world and functional components were just a bold idea, until recent times, when a new paradigm of State has emerged: Async State.

Local State

Alright, everyone who has already worked with React knows what a Local State is.

I don't know what it is
Local State is the state of a single Component.

Every time a state is updated, the component re-renders.

You may have worked with this ancient structure:

class CommitList extends React.Component {
  constructor(props) {
    super(props);
    this.state = {
      isLoading: false,
      commits: [],
      error: null
    };
  }

  componentDidMount() {
    this.fetchCommits();
  }

  fetchCommits = async () => {
    this.setState({ isLoading: true });
    try {
      const response = await fetch('https://api.github.com/repos/facebook/react/commits');
      const data = await response.json();
      this.setState({ commits: data, isLoading: false });
    } catch (error) {
      this.setState({ error: error.message, isLoading: false });
    }
  };

  render() {
    const { isLoading, commits, error } = this.state;

    if (isLoading) return <div>Loading...</div>;
    if (error) return <div>Error: {error}</div>;

    return (
      <div>
        <h2>Commit List</h2>
        <ul>
          {commits.map(commit => (
            <li key={commit.sha}>{commit.commit.message}</li>
          ))}
        </ul>
        <TotalCommitsCount count={commits.length} />
      </div>
    );
  }
}

class TotalCommitsCount extends Component {
  render() {
    return <div>Total commits: {this.props.count}</div>;
  }
}
}

Perhaps a modern functional one:

const CommitList = () => {
  const [isLoading, setIsLoading] = useState(false);
  const [commits, setCommits] = useState([]);
  const [error, setError] = useState(null);

  // To update state you can use setIsLoading, setCommits or setUsername.
  // As each function will overwrite only the state bound to it.
  // NOTE: It will still cause a full-component re-render
  useEffect(() => {
    const fetchCommits = async () => {
      setIsLoading(true);
      try {
        const response = await fetch('https://api.github.com/repos/facebook/react/commits');
        const data = await response.json();
        setCommits(data);
        setIsLoading(false);
      } catch (error) {
        setError(error.message);
        setIsLoading(false);
      }
    };

    fetchCommits();
  }, []);

  if (isLoading) return <div>Loading...</div>;
  if (error) return <div>Error: {error}</div>;

  return (
    <div>
      <h2>Commit List</h2>
      <ul>
        {commits.map(commit => (
          <li key={commit.sha}>{commit.commit.message}</li>
        ))}
      </ul>
      <TotalCommitsCount count={commits.length} />
    </div>
  );
};

const TotalCommitsCount = ({ count }) => {
  return <div>Total commits: {count}</div>;
};

Or even a "more accepted" one? (Definitely more rare though)

const initialState = {
  isLoading: false,
  commits: [],
  userName: ''
};

const reducer = (state, action) => {
  switch (action.type) {
    case 'SET_LOADING':
      return { ...state, isLoading: action.payload };
    case 'SET_COMMITS':
      return { ...state, commits: action.payload };
    case 'SET_USERNAME':
      return { ...state, userName: action.payload };
    default:
      return state;
  }
};

const CommitList = () => {
  const [state, dispatch] = useReducer(reducer, initialState);
  const { isLoading, commits, userName } = state;

  // To update state, use dispatch. For example:
  // dispatch({ type: 'SET_LOADING', payload: true });
  // dispatch({ type: 'SET_COMMITS', payload: [...] });
  // dispatch({ type: 'SET_USERNAME', payload: 'newUsername' });
};

Which can make you wonder...

Why the hack would I be writing this complex reducer for a single component?

Well, React inherited this ugly hook called useReducer from a very important tool called Redux.

If you ever had to deal with Global State Management in React, you must've heard about Redux.

This brings us to the next topic: Global State Management.

Global State

Global State Management is one of the first complex subjects when learning React.

What is it?

It can be multiple things, built in many ways, with different libraries.

I like to define it as:

A single JSON object, accessed and maintained by any Component of the application.

const globalState = { 
  isUnique: true,
  isAccessible: true,
  isModifiable: true,
  isFEOnly: true
}

I like to think of it as:

A Front-End No-SQL Database.

That's right, a Database. It's where you store application data, that your components can read/write/update/delete.

I know, by default, the state will be recreated whenever the user reloads the page, but that may not be what you want it to do, and if you're persisting data somewhere (like the localStorage), you might want to learn about migrations to avoid breaking the app every new deployment.

I like to use it as:

A multidimensional portal, where components can dispatch their feelings and select their attributes. Everything, everywhere, all at once.

How to use it?

The main way

Redux

It is the industry standard.

I have worked with React, TypeScript, and Redux for 7 years. Every project I've worked with professionally uses Redux.

The vast majority of people I've met who works with React, use Redux.

The most mentioned tool in React open positions at Trampar de Casa is Redux.

The most popular React State Management tool is...

Redux

If you want to work with React, you should learn Redux.
If you currently work with React, you probably already know.

Ok, here's how we usually fetch data using Redux.

Disclaimer
"What? Does this make sense? Redux is to store data, not to fetch, how the F would you fetch data with Redux?"

If you thought about this, I must tell you:

I'm not actually fetching data with Redux.
Redux will be the cabinet for the application, it'll store ~shoes~ states that are directly related to fetching, that's why I used this wrong phrase: "fetch data using Redux".

// actions
export const SET_LOADING = 'SET_LOADING';
export const setLoading = (isLoading) => ({
  type: SET_LOADING,
  payload: isLoading,
});

export const SET_ERROR = 'SET_ERROR';
export const setError = (isError) => ({
  type: SET_ERROR,
  payload: isError,
});

export const SET_COMMITS = 'SET_COMMITS';
export const setCommits = (commits) => ({
  type: SET_COMMITS,
  payload: commits,
});


// To be able to use ASYNC action, it's required to use redux-thunk as a middleware
export const fetchCommits = () => async (dispatch) => {
  dispatch(setLoading(true));
  try {
    const response = await fetch('https://api.github.com/repos/facebook/react/commits');
    const data = await response.json();
    dispatch(setCommits(data));
    dispatch(setError(false));
  } catch (error) {
    dispatch(setError(true));
  } finally {
    dispatch(setLoading(false));
  }
};

// the state shared between 2-to-many components
const initialState = {
  isLoading: false,
  isError: false,
  commits: [],
};

// reducer
export const rootReducer = (state = initialState, action) => {
  // This could also be actions[action.type].
  switch (action.type) {
    case SET_LOADING:
      return { ...state, isLoading: action.payload };
    case SET_ERROR:
      return { ...state, isError: action.payload };
    case SET_COMMITS:
      return { ...state, commits: action.payload };
    default:
      return state;
  }
};

Now on the UI side, we integrate with actions using useDispatch and useSelector:

// Commits.tsx
import React, { useEffect } from 'react';
import { useDispatch, useSelector } from 'react-redux';
import { fetchCommits } from './action';

export const Commits = () => {
  const dispatch = useDispatch();
  const { isLoading, isError, commits } = useSelector(state => state);

  useEffect(() => {
    dispatch(fetchCommits());
  }, [dispatch]);

  if (isLoading) return <div>Loading...</div>;
  if (isError) return <div>Error while trying to fetch commits.</div>;

  return (
    <ul>
      {commits.map(commit => (
        <li key={commit.sha}>{commit.commit.message}</li>
      ))}
    </ul>
  );
};

If Commits.tsx was the only component that needed to access commits list, you shouldn't store this data on the Global State. It could use the local state instead.

But let's suppose you have other components that need to interact with this list, one of them may be as simple as this one:

// TotalCommitsCount.tsx
import React from 'react';
import { useSelector } from 'react-redux';

export const TotalCommitsCount = () => {
  const commitCount = useSelector(state => state.commits.length);
  return <div>Total commits: {commitCount}</div>;
}

Disclaimer
In theory, this piece of code would make more sense living inside Commits.tsx, but let's assume we want to display this component in multiple places of the app and it makes sense to put the commits list on the Global State and to have this TotalCommitsCount component.

With the index.js component being something like this:

import React from 'react';
import ReactDOM from 'react-dom';
import thunk from 'redux-thunk';
import { createStore, applyMiddleware } from 'redux';
import { Provider } from 'react-redux';
import { Commits } from "./Commits"
import { TotalCommitsCount } from "./TotalCommitsCount"

export const App = () => (
    <main>
        <TotalCommitsCount />
        <Commits />
    </main>
)

const store = createStore(rootReducer, applyMiddleware(thunk));
ReactDOM.render(
  <Provider store={store}>
    <App />
  </Provider>,
  document.getElementById('root')
);

This works, but man, that looks overly complicated for something as simple as fetching data right?

Redux feels a little too bloated to me.

You're forced to create actions and reducers, often also need to create a string name for the action to be used inside the reducer, and depending on the folder structure of the project, each layer could be in a different file.

Which is not productive.

But wait, there is a simpler way.

The simple way

Zustand

At the time I'm writing this article, Zustand has 3,495,826 million weekly downloads, more than 45,000 stars on GitHub, and 2, that's right, TWO open Pull Requests.

ONE OF THEM IS ABOUT UPDATING IT'S DOC

If this is not a piece of Software Programming art, I don't know what it is.

Here's how to replicate the previous code using Zustand.

// store.js
import create from 'zustand';

const useStore = create((set) => ({
  isLoading: false,
  isError: false,
  commits: [],
  fetchCommits: async () => {
    set({ isLoading: true });
    try {
      const response = await fetch('https://api.github.com/repos/facebook/react/commits');
      const data = await response.json();
      set({ commits: data, isError: false });
    } catch (error) {
      set({ isError: true });
    } finally {
      set({ isLoading: false });
    }
  },
}));

This was our Store, now the UI.

// Commits.tsx
import React, { useEffect } from 'react';
import useStore from './store';

export const Commits = () => {
  const { isLoading, isError, commits, fetchCommits } = useStore();

  useEffect(() => {
    fetchCommits();
  }, [fetchCommits]);

  if (isLoading) return <div>Loading...</div>;
  if (isError) return <div>Error occurred</div>;

  return (
    <ul>
      {commits.map(commit => (
        <li key={commit.sha}>{commit.commit.message}</li>
      ))}
    </ul>
  );
}

And last but not least.

// TotalCommitsCount.tsx
import React from 'react'; 
import useStore from './store'; 
const TotalCommitsCount = () => { 
    const totalCommits = useStore(state => state.commits.length);
    return ( 
        <div> 
            <h2>Total Commits:</h2> <p>{totalCommits}</p> 
        </div> 
    ); 
};

There are no actions and reducers, there is a Store.

And it's advisable to have slices of Store, so everything is near to the feature related to the data.

It works perfect with a folder-by-feature folder structure.

The wrong way

I need to confess something, both of my previous examples are wrong.

And let me do a quick disclaimer: They're not wrong, they're outdated, and therefore, wrong.

This wasn't always wrong though. That's how we used to develop data fetching in React applications a while ago, and you may still find code similar to this one out there in the world.

But there is another way.

An easier one, and more aligned with an essential feature for web development: Caching. But I'll get back to this subject later.

Currently, to fetch data in a single component, the following flow is required:

What happens if I need to fetch data from 20 endpoints inside 20 components?

20x isLoading + 20x isError + 20x actions to mutate this properties.

What will they look like?

With 20 endpoints, this will become a very repetitive process and will cause a good amount of duplicated code.

What if you need to implement a caching feature to prevent recalling the same endpoint in a short period? (or any other condition)

Well, that will translate into a lot of work for basic features (like caching) and well-written components that are prepared for loading/error states.

This is why Async State was born.

Async State

Before talking about Async State I want to mention something. We know how to use Local and Global state but at this time I didn't mention what should be stored and why.

The Global State example has a flaw and an important one.

The TotalCommitsCount component will always display the Commits Count, even if it's loading or has an error.

If the request failed, there's no way to know that the Total Commits Count is 0, so presenting this value is presenting a lie.

In fact, until the request finishes, there is no way to know for sure what's the Total Commits Count value.

This is because the Total Commits Count is not a value we have inside the application. It's external information, async stuff, you know.

We shouldn't be telling lies if we don't know the truth.

That's why we must identify Async State in our application and create components prepared for it.

We can do this with React-Query, SWR, Redux Toolkit Query and many others.

For this article, I'll use React-Query.

I recommend you to access the docs of each of these tools to better understand which problems they solve.

Here's the code:

No more actions, no more dispatches, no more Global State for fetching data.

This is what you have to do in your App.tsx file to have React-Query properly configured:

You see, Async State is special.

It's like Schrödinger's cat – you don't know the state until you observe it (or run it).

But wait, if both components are calling useCommits and useCommits is calling an API endpoint, does this mean that there will be TWO identical requests to load the same data?

Short Answer: no!

Long Answer: React Query is awesome. It automatically handles this situation for you, it comes with pre-configured caching that is smart enough to know when to refetch your data or simply use the cache.

It's also extremely configurable so you can tweak to fit 100% of your application's needs.

Now we have our components always ready for isLoading or isError and we keep the Global State less polluted and have some pretty neat features out-of-the-box.

Conclusion

Now you know the difference between Local, Global and Async State.

Local -> Component Only.
Global -> Single-Json-NoSQL-DB-For-The-FE.
Async -> External data, Schrodinger's cat-like, living outside of the FE application that requires Loading and can return Error.

I hope you enjoyed this article, let me know if you have different opinions or any constructive feedback, cheers!

Node vs Go: API Showdown

Caio Borghi — Mon, 01 Jan 2024 14:15:57 +0000

Disclaimer and Introduction
How were the metrics gathered?
- Tech Stack
- Flow Diagram
- Manual RDS Setup
- Environment Initialization
- Monitoring Process
- Script Parameters
2,000 Requests per Second
- Latency x Seconds
- File Descriptors Count
- Threads Count
- RAM
- CPU
- Overall
3,000 Requests per Second
- Dev.to can't link to same-name sections 😔
5,000 Requests per Second
- But they exist, and they're awesome!
Final Considerations

Disclaimer and Introduction

This blog is primarily for fun and educational exploration. The results here should not be the sole basis of your technical decisions. It does not mean that one language is better than the other, please do not read it so seriously.

In fact, it does not make much sense to compare such different languages.

Cool, with that being said, let's have some fun, compare some metrics, and get a better understanding of how both languages deal with some key aspects (RAM, CPU, Open File Descriptors Count & OS Threads Count) when under severe pressure.

How were the metrics gathered?

If you want to know how this benchmark was profiled, expand the below section, otherwise, you can skip directly to the results 🤓.

Behind the scenes

Tech Stack

It was created using the following technologies:

1x EC2 t2.micro (the API's command center)
1x EC2 t2.xlarge (the request-launching gun)
A Postgres RDS for data persistence
Vegeta for unleashing HTTP load
Golang 1.21.4 and Node.js 21.4.0 for the API showdown
Open Tofu for automagically spinning up our servers

Important: The API server has only a 1-core processor with 1GB of RAM. This post shows the efficiency of both approaches in a very limited environment. You can check the full code here.

Flow Diagram

How does the communication happen?

Both servers are located within the same VPC in AWS, ensuring minimal latency. However, the RDS, although situated in the same AWS Region (sa-east-1), operates in another VPC, introducing a more realistic latency.

This is good because, in a real-world scenario, there will be latency.

Manual RDS Setup

Unfortunately, I wasn't able to set up the Postgres RDS with OpenTofu, (cough cough: skill issue) so I had to manually craft it on AWS and execute the following script:

CREATE TABLE IF NOT EXISTS users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) NOT NULL,
    password VARCHAR(255) NOT NULL
);
TRUNCATE TABLE users;

Ok, with everything in place, it's showtime!

Environment Initialization

Start the environment with:

tofu apply -auto-approve

Full main.tf file here

What does it do?

Launches 1 VPC + 2 Subnets
Boots up 2 Ubuntu Servers, executing specific scripts
Install Node + Golang on the API server
Sets up Vegeta on the Gun server
Deploys the API and load-tester code
Generates 2 SSH scripts for connectivity (ssh_connect_api.sh & ssh_connect_gun.sh)

Monitoring Process

With this setup, I can access the API server and initiate either the Node or Go API.

Concurrently, start the monitor_process.sh to snag metrics like RAM, CPU, Threads Count, & File Descriptors Count and save to a .csv file.

All is done based on the process ID of the running API.

Check the script here!

Script Parameters

Process ID
The number of requests per second (to name the CSV file correctly)

Once the API is running, I get the process ID using console.log(process.pid) on Node or fmt.Printf("ID: %d", os.Getpid()) on Golang.

Then, I can simply run:

./monitor_process.sh 2587 2000

This command monitors our process, updating a .csv file named process_stats_2000.csv every second with fresh data.

Ok, now let's analyze the results, compare both APIs, and see what learnings can we squeeze from it, let's get started!

2,000 Requests per Second

Alright, for this first step, I ran this Vegeta script that fires 2,000 requests per second over 30s to the Server API.

This was done inside the Gun Server by running

./metrics.sh 2000

Which produces the following output:

Starting Vegeta attack for 30s at 2000 requests per second...

Then, I combined the results in some beautiful charts, let's take a look at them:

Latency x Seconds

By looking at the latency chart, we can see that Golang struggled a lot initially, taking ~5s to stabilize.

This may have been a one-time anomaly, but as I won't redo the test and the metrics are all correct, I'll call this one a lucky shot for Node.

Node kept a consistent latency throughout most of the test, with spikes at 12s and 20s.

Golang, on the other hand, had some trouble stabilizing the latency at the beginning, costing the pole position. However, It went well after that, by keeping the latency around 230ms.

File Descriptors Count

This one is interesting.

In Linux, a new socket and a corresponding File Descriptor (FD) are created for each incoming server connection. These FDs store connection information.

On Ubuntu, the default soft limit for open file descriptors is 1024.

However, both Go and Node ignore the soft limit and always use the hard limit. This can be verified by accessing the /proc/$PID/limits FD after the node/go process has started.

You can use the command ulimit -n to see the OS soft limit of open file descriptors of the current shell session.

Ok then, this means that the OS does not interfere with the number of open FDs; the programming language manages it.

In this test, Node kept a lower, but irregular, number of open FDs while Golang spiked until 8.000, stabilized, and remained consistent until the end.

Threads Count

Wasn't Node.js single-threaded? 🤯

Well, no.

By default, Node starts a few threads:

1 Main Thread: Executes JavaScript code and handles the event loop.
4 Worker Threads (default libuv thread pool)
- Handles blocking async I/O such as DNS lookup queries, crypto module, and some file I/O operations.
V8 Threads:
- 1 Compiler Thread: Compiles JavaScript into native machine code.
- 1 Profiler Thread: Collects performance profiles for optimizations.
- 2 or more Garbage Collector Threads: Manages memory allocation and garbage collection.
Additional Internal Threads: Number varies, for various Node.js and V8 background tasks.

I noticed that, at the startup of the Node Process, it created 11 OS threads and once the requests started arriving, the count jumped to 15 OS threads and stayed there.

Go, on the other hand, kept 4 stable OS threads.

RAM

TL;DR

Node.js:
- Consistently low RAM usage, between 75MB and 120MB.
- Utilizes an Event-Loop for I/O operations, avoiding new threads.
- More about Node's Event Loop: Exploring Asynchronous I/O in Node.js.
Go:
- Higher initial RAM usage, stabilizing at 300MB.
- Spawns a new goroutine for each network request.
- Goroutines are lighter than OS threads but still impact memory under load.
- Insight into Go's Runtime scheduler: Go Runtime Scheduler Talk.

Explanation
Node kept a lower and more stable RAM usage, between 75MB and 120MB, throughout the test.

Meanwhile, Go's RAM usage increased in the first seconds until it stabilized at 300MB (almost tripling Node's peak).

This difference can be explained due to how both languages deal with asynchronous operations, like I/O database communication.

Node uses an Event-Loop approach, which means it doesn't create new thread. In contrast, Go often spawns a new goroutine for each request, which increases memory usage. The goroutine is a lightweight thread managed by the Go Runtime.

Even though lighter than an OS Thread, it still leaves a memory footprint when under heavy load.

For insights on the Node Event Loop, check this blog post I wrote.

To better understand the Go Runtime scheduler, please watch this phenomenal talk - one of the best I've ever watched.

CPU

Node was able to use less CPU than Go at this one, this may be because the Go Runtime is more complex and requires more steps/calculations than the libuv's Event Loop.

Overall

I must be honest: I was surprised by this result.

Node won this one 🏆.

It showcased:

Superior p99 latency, responding in under 1.2s for 99% of requests, compared to Go's 4.31s
Faster average latency, clocking in at 147ms versus Go's 459ms, 3.1x faster!
Significantly smaller maximum latency, peaking at just 1.5s against Go's 6.4s, which was 4.2 slower. (c'mon Gopher, you're looking bad!)

3,000 Requests per Second

Now let's redo the test, send 3,000 requests/s over 30s for each API, and see the results.

Latency x Seconds

While Go was able to keep a really stable latency with only two small spikes, Node was in some deep trouble and showcased a very inconsistent latency throughout the test.

File Descriptors Count

Remember I told you that neither Node nor Go respects the soft limit of the Open File Descriptors and both languages manage it by themselves?

Here's a fun fact:

Golang was able to process, handle, and deliver more requests, in a shorter time, using fewer resources by setting a "hard" limit of open FDs at each period of the test (based on some metric that I'm not sure which one).

This is super cool!

Look at how Go managed its FDs:

8 FDs: In the first 0-3 seconds
1,590 FDs: Between 4-17 seconds
2,225 FDs: Between 18-31 seconds

Node, on the other hand, didn't interfere with the open file descriptors like Go did. You can see that on the chart.

Empirically, it seems that Go is pre-allocating (or pre-opening) File Descriptors at some rate and reusing them instead of generating one for each connection at the time they arrive.

I'm not sure exactly how they do that, though, feel free to comment if you have some hint 😄

I found some good reads about this:

Does net/http have connection pool? - talks about Go's net/http package and how it manages connections.
The Go Netpoller - article that explains about the Go Netpoller.

Threads Count

Ok, something worth noticing happened on this test.

Node: peaked from 11 to 15 OS Threads as the requests started arriving. I believe this is due to the DNS Lookup operations, as briefly mentioned in this issue.

Go: Stepped up its game from 4 to 5 OS Threads. It's the Runtime Scheduler orchestrating the show, Go is smart enough to pack multiple Goroutines into each OS Thread. When it gets clumsy, it smoothly starts a new OS thread.

This approach is not just efficient; it's a masterclass in resource optimization, squeezing every last bit of performance from the hardware. It is Amazing! 🚀

RAM

At this time, you probably noticed that the Node.js line lasts longer than the Go line, well, this is because the API took more time to answer all the requests it received.

This has also impacted the RAM usage. Remember that for the first test Node's RAM usage was way below Go's one?

That's not the case when you have tons of connections hanging on the server waiting to be processed.

CPU

This time, Node required much more CPU Usage than that and was able to keep the usage below 35% while Node peaked at 64%.

Overall

🎉 We have a fight! 🎉

The dispute is open and Golang was way superior at this one, let's look at the numbers:

Golang had:

Lower Latencies
- p99: 738.873ms against 30.001s, 40 times lower than Node.
- Average: 60.454ms versus 7.079s - 118 times faster
- Maximum: Go peaked at 1.33s, while Node reached the sky with 30.0004s.
Perfect Success Rate (100%)
- Against 91.93% from Node, which had some requests failing.

That was a massacre, it was like comparing a new sports car with a Fusquinha.

Detailed comparison

Node.js Performance Metrics:

Total Requests: 86,922 with a rate of 2,897.33 per second.
Throughput: 1,449.29 requests per second.
Duration:
- Total: 55.135 seconds.
- Attack Phase: 30.001 seconds.
- Wait Time: 25.134 seconds.
Latencies:
- Minimum: 3.458 ms.
- Mean: 7.079 seconds.
- Median (50th Percentile): 6.068 seconds.
- 90th Percentile: 9.563 seconds.
- 95th Percentile: 26.814 seconds.
- 99th Percentile: 30.001 seconds.
- Maximum: 30.004 seconds.
Data Transfer:
- Bytes In: 2,077,556 (average 23.90 bytes/request).
- Bytes Out: 7,351,352 (average 84.57 bytes/request).
Success Ratio: 91.93%.
Status Codes: 7016 failures, 79,906 successes (201 code).

Golang Performance Metrics:

Total Requests: 90,001 with a rate of 3,000.09 per second.
Throughput: 2,999.89 requests per second.
Duration:
- Total: 30.001 seconds.
- Attack Phase: 29.999 seconds.
- Wait Time: 2.035 ms.
Latencies:
- Minimum: 1.371 ms.
- Mean: 60.454 ms.
- Median (50th Percentile): 4.773 ms.
- 90th Percentile: 194.115 ms.
- 95th Percentile: 453.031 ms.
- 99th Percentile: 736.873 ms.
- Maximum: 1.33 seconds.
Data Transfer:
- Bytes In: 2,430,027 (average 27.00 bytes/request).
- Bytes Out: 8,280,092 (average 92.00 bytes/request).
Success Ratio: 100%.
Status Codes: All 90,001 requests were successful (201 code).
Throughput: Go had a higher throughput compared to Node.
Latencies: Node exhibited significantly higher latencies, especially in the mean, 95th, and 99th percentiles.
Success Rate: Go achieved a 100% success rate, whereas Node had a lower success rate with some failed requests.

5,000 Requests per second

Final round, let's see how both languages deal with severe pressure.

Latency x Seconds

Go was able to keep a very low, stable latency until ~20 seconds, when it started to present some troubles, that caused peaks of 5s, which is very slow.

Node presented problems throughout the entire test, responding with latencies between 5-10s.

It's nice to notice that even in a very stressful test, Go could be stable over the entire test.

File Descriptors Count

Once again we can see how stable are the Open File Descriptors of Golang versus how unmanaged, linear-growing they are for Node.js

I believe that this is directly related to the Go Network Poller that reuses (and maybe pre-creates) File Descriptors instead of creating one at the time each request arrives.

I wonder if Node could benefit from such an approach, will definitely check this out 😅

Threads Count

In this chart we can see that Node started with 11 OS Threads and jumped to 15 whenever connections started arriving, while Golang kept 4 OS threads for the majority of the test, increasing to 5 at the end.

Go's strategy seems to be more stable under heavy loads.

RAM

Node.js showed a linear increase in RAM usage, whereas Go's increase was step-like, similar to climbing a ladder.

This pattern in Go is due to its runtime actively managing resources and setting limits for go routines, OS threads, and open file descriptors.

CPU

The CPU usage pattern is very similar for both languages, suggesting that this may be outside of the language control, being delegated to the OS.

Overall

Go excels again with a higher Success Rate and lower p99, average, min, and max latency.

Given that Go is a compiled language, and Node.js (JavaScript) is interpreted, this outcome is expected.
Compiled languages typically have fewer steps before executing machine code.

Despite its inherent challenges, Node.js managed to successfully process 89.38% of the requests.

Final Considerations

Thank you for taking the time to read this blog post 🙏

It's no surprise that Go, a compiled language focused on concurrency and parallelism by design, came out on top. Still, it was interesting to see how it all played out.

It was cool to see how Go and Node.js handle tasks differently and how that impacts the computer's resources.

I've summed up the key points below.

Open File Descriptor Management

Go: Demonstrates a strategy of pre-allocation and reuse for File Descriptors, thanks to its intelligent network poller and resource management. This approach contributes to efficient handling and scalability under heavy network loads.
Node.js: Shows a dynamic, maybe unmanaged pattern in File Descriptor usage, reflecting its approach to handling server connections and opening FDs one by one.

Thread Management and Node.js

Go: Maintains a stable, low OS thread count, highlighting the efficiency of its runtime scheduler in optimizing thread usage, especially under heavy stress 🤯.
Node.js: Contrary to popular belief, Node.js uses multiple threads for tasks like DNS lookups, Garbage Collector (Hi V8), and blocking async I/O ops, it's not just a single thread.

Inside Node.js: Exploring Asynchronous I/O

Caio Borghi — Mon, 11 Dec 2023 18:22:39 +0000

Introduction
How Node Handles Asynchronous Code
Asynchronous Operations: What Are They?
- Blocking vs Non-Blocking Asynchronous Operation
Experiments with Blocking Functions
Experiments with Non-Blocking Functions
Non-Blocking Asynchronous Operations and OS
- Understanding File Descriptors
  - What is a FD?
  - FD and Non-Blocking I/O
- Monitoring FDs with syscalls
  - Understanding select
  - Epoll
  - io_uring
Conclusion

Introduction

Recently, I've been studying asynchronous code execution in Node.js.

I ended up learning (and writing) a lot, from an article about how the Event Loop works to a Twitter thread explaining who waits for the http request to finish.

If you want, you can also access the mind map I created before writing this post by clicking here.

Now, let's get to the point!

How Node Handles Asynchronous Code

In Node:

All JavaScript code is executed in the main thread.
The libuv library is responsible for handling I/O (In/Out) operations, i.e., asynchronous operations.
By default, libuv provides 4 worker threads for Node.js.
- These threads will only be used when blocking asynchronous operations are performed, in which case they will block one of the libuv threads (which are OS threads) instead of the main Node execution thread.
There are both blocking and non-blocking operations, and most of the current asynchronous operations are non-blocking.

Asynchronous Operations: What Are They?

Generally, there's confusion when it comes to asynchronous operations.

Many believe it means something happens in the background, in parallel, at the same time, or in another thread.

In reality, an asynchronous operation is an operation that won't return now, but later.

They depend on communication with external agents, and these agents might not have an immediate response to your request.

We're talking about I/O (input/output) operations.

Examples:

Reading a file: data leaves the disk and enters the application.
Writing to a file: data leaves the application and enters the disk.
Network Operations
- HTTP requests, for example.
- The application sends an http request to some server and receives the data.

Blocking vs Non-Blocking Asynchronous Operation

In the modern world, ~~people don't talk to each other~~ most asynchronous operations are non-blocking.

But wait, does that mean:

libuv provides 4 threads (by default).
they "take care of" the blocking I/O operations.
the vast majority of operations are non-blocking.

Seems kind of useless, right?

With this question in mind, I decided to do some experiments.

Experiments with Blocking Functions

First, I tested an asynchronous CPU-intensive function, one of the rare asynchronous blocking functions in Node.

The used code was as follows:

// index.js
import { pbkdf2 } from "crypto";

const TEN_MILLIONS = 1e7;

// CPU-intensive asynchronous function
// Goal: Block a worker thread
// Original goal: Generate a passphrase
// The third parameter is the number of iterations
// In this example, we are passing 10 million
function runSlowCryptoFunction(callback) {
  pbkdf2("secret", "

salt", TEN_MILLIONS, 64, "sha512", callback);
}

// Here we want to know how many worker threads libuv will use
console.log(`Thread pool size is ${process.env.UV_THREADPOOL_SIZE}`);

const runAsyncBlockingOperations = () => {
  const startDate = new Date();
  const runAsyncBlockingOperation = (runIndex) => {
    runSlowCryptoFunction(() => {
      const ms = new Date() - startDate;
      console.log(`Finished run ${runIndex} in ${ms/1000}s`);
    });
  }
  runAsyncBlockingOperation(1);
  runAsyncBlockingOperation(2);
};

runAsyncBlockingOperations();

To validate the operation, I ran the command:

UV_THREADPOOL_SIZE=1 node index.js

IMPORTANT:

UV_THREADPOOL_SIZE: It's an environment variable that determines how many libuv worker threads Node will start.

The result was:

Thread pool size is 1
Finished run 1 in 3.063s
Finished run 2 in 6.094s

That is, with only one thread, each execution took ~3 seconds, and they occurred sequentially. One after the other.

Now, I decided to do the following test:

UV_THREADPOOL_SIZE=2 node index.js

And the result was:

Thread pool size is 2
Finished run 2 in 3.225s
Finished run 1 in 3.243s

With that, it's proven that LIBUV's Worker Threads in Node.js handle blocking asynchronous operations.

But what about the non-blocking ones? If no one waits for them, how do they work?

I decided to write another function to test it.

Experiments with Non-Blocking Functions

The fetch function (native to Node) performs a non-blocking network asynchronous operation.

With the following code, I redid the test of the first experiment:

//non-blocking.js
// Here we want to know how many worker threads libuv will use
console.log(`Thread pool size is ${process.env.UV_THREADPOOL_SIZE}`);

const startDate = new Date();
fetch("https://www.google.com").then(() => {
  const ms = new Date() - startDate;
  console.log(`Fetch 1 returned in ${ms / 1000}s`);
});

fetch("https://www.google.com").then(() => {
  const ms = new Date() - startDate;
  console.log(`Fetch 2 returned in ${ms / 1000}s`);
});

And I executed the script with the following command:

UV_THREADPOOL_SIZE=1 node non-blocking.js

The result was:

Thread pool size is 1
Fetch 1 returned in 0.391s
Fetch 2 returned in 0.396s

So, I decided to test with two threads, to see if anything changed:

UV_THREADPOOL_SIZE=2 node non-blocking.js

And then:

Thread pool size is 2
Fetch 2 returned in 0.402s
Fetch 1 returned in 0.407s

With this, I observed that:

Having more threads running in LIBUV does not help in the execution of non-blocking asynchronous operations.

But then, I questioned again, if no libuv thread is "waiting" for the request to return, how does this work?

My friend, that's when I fell into a gigantic hole of research and knowledge about the operation of:

Non-Blocking Asynchronous Operations and OS

The Operating System has evolved quite a bit over the years to deal with non-blocking I/O operations, this is done through syscalls, they are:

select/poll: These are the traditional ways of dealing with non-blocking I/O and are generally considered less efficient.
IOCP: Used in Windows for asynchronous operations.
kqueue: A method for MacOS and BSD.
epoll: Efficient and used in Linux. Unlike select, it is not limited by the number of FDs.
io_uring: An evolution of epoll, bringing performance improvements and a queue-based approach.

To understand better, we need to dive into the details of non-blocking I/O operations.

Understanding File Descriptors

To explain non-blocking I/O, I need to quickly explain the concept of File Descriptors (FDs).

What is a FD?

It's a numerical index of a table maintained by the kernel, where each record has:

Resource type (such as file, socket, device).
Current position of the file pointer.
Permissions and flags, defining modes like read or write.
Reference to the resource's data structure in the kernel.

They are fundamental for I/O management.

FD and Non-Blocking I/O

When initiating a non-blocking I/O operation, Linux associates an FD with it without interrupting (blocking) the process's execution.

For example:

Imagine you want to read the contents of a very large file.

Blocking approach:

The process calls the read file function.
The process waits while the OS reads the file's content.
- The process is blocked until the OS finishes.

Non-blocking approach:

The process requests asynchronous read.
The OS starts reading the content and returns an FD to the process.
The process isn't locked up and can do other things.
Periodically, the process calls a syscall to check if the reading is finished.

The process decides the mode of reading through the fcntl function with the O_NONBLOCK flag, but this is secondary at the moment.

Monitoring FDs with syscalls

To efficiently observe multiple FDs, OSs rely on some syscalls:

Understanding select:

Receives a list of FDs.
Blocks the process until one or more FDs are ready for the specified operation (read, write, exception).
After the syscall returns, the program can iterate over the FDs to identify those ready for I/O.
Uses a search algorithm that is O(n).
- Inefficient, slow, tiresome with many FDs.

Epoll

An evolution of select, it uses a self-balancing tree to store the FDs, making access time almost constant, O(1).

Pretty fancy!

How it works:

Create an epoll instance with epoll_create.
Associate the FDs with this instance using epoll_ctl.
Use epoll_wait to wait for activity on any of the FDs.
Has a timeout parameter.
- Extremely important and well utilized by the libuv Event Loop!

Io_uring

This is a game-changer.

While epoll significantly improved the performance of searching and handling FDs, io_uring rethinks the entire nature of I/O operations.

And so, after understanding how it works, I wondered how nobody thought of this before!!!

Recapping:

select: Receives a list of FDs, stores them sequentially (like an array), and checks each one for changes or activity, with complexity O(n).
epoll: Receives a list of FDs, stores them using a self-balancing tree, does not check each one individually, is more efficient, and does the same as select but with complexity O(1).

Historically, the process was responsible for iterating over the returned FDs to know which have finished or not.

io_uring: What? Return a list? Do polling? Are you kidding? Ever heard of queues?

It works using two main queues, in the form of rings (hence the name io-ring).

1 for submitting tasks.
1 for completed tasks.

Simple, right?

The process, when starting an I/O operation, queues the operation using the io_uring structure.

Then, instead of calling select or epoll and iterating over the returned FDs, the process can choose to be notified when an I/O operation is completed.

Polling? No. Queues!

Conclusion

With this knowledge, I now understand precisely the path Node takes to perform an asynchronous operation.

If it's blocking:

Executes the asynchronous operation using libuv.
Adds it to a libuv worker thread.
The worker thread is blocked, waiting for the operation to finish.
Once finished, the thread is responsible for placing the result in the Event Loop in the MacroTasks queue.
The callback is executed on the main thread.

If it's non-blocking:

Executes the asynchronous operation using libuv.
Libuv performs a non-blocking I/O syscall.
Performs polling with the FDs until they resolve (epoll).
From version 20.3.0, uses io_uring.
- Queue-based approach for submission/completed operations.
Upon receiving the event of operation completion:
- libuv takes care of executing the callback on the main thread.

A Deep Dive into Green Threads and Node.js

Caio Borghi — Sun, 10 Dec 2023 13:01:51 +0000

Hey, how's it going?

Recently, I've been studying concurrency and parallelism and came across a nice thing called Green Threads.

In this post, I'll explain what they are, and how I tried implementing them on Node.js and failed miserably, hope you enjoy it!

Prelude
Processes
- Starting a Process
Threads
Green/Virtual Threads
- Implementing Green Threads
- Preemptive Scheduler
- Cooperative Scheduler
Conclusion

Prelude

Before explaining what are green threads, we must first understand what is a thread.

To be honest, to be able to explain what those are, I'll need to talk about Processes.

Processes

What is a process?

A process is the live version of a computer program.

As if the code you write were a recipe and the process you execute was the execution or process to make the recipe real.

So... What is a process?

It's the execution of a program. Your browser, text editor, image preview, and file explorer, they're all processes. More often than not, they start not one but multiple processes.

How to start a process?

If you're a Windows user, by double-clicking some application or executing them on the CMD (Command Prompt) or Windows Terminal.

If you're a Linux/MacOS user, you can also use the slow-but-common double-click to open processes or, you can pretend to be a series tv hacker and use the terminal.

Behind the hood, whenever you start a process:

The OS finds an unused section of main memory that is large enough for the application.
The OS makes a copy of the application and its data in that section of the main memory.
The OS sets up resources for the application.
Finally, the OS starts the application.

Good to know: A process consumes memory and has instructions that'll be executed by the, oh no, the Processor.

That's right, the Processor (aka CPU) processes instructions from a process and that requires memory.

Each process has its own space in memory.

A process cannot access another process's memory, even though it can communicate through messages sometimes.

Threads

What about threads?

Threads are lightweight processes.

A process can start many threads, and a thread shares the parent-process memory.

Meaning that two threads started by the same process could potentially access the same memory.

OS Threads have a cost: memory.

They're lighter than a new process but still require memory to be created, it may not seem too much, but if you take a look at this chart:

You can see that one approach (Apache) consumes a lot more memory than another (Nginx).

This slide was presented at NodeJS's first talk ever.

I strongly recommend you to watch, it's awesome!

Apache consumes more memory because it uses a one-thread-per-request approach while Nginx uses a non-blocking event-loop approach to handle new requests, i.e. no threads.

Ok then, threads are lightweight processes but still consume memory and when you need to scale (to multiple threads) they start to get costy.

How to solve that?

Well, if you don't want to use an event loop with non-blocking I/O, there is a way: Green Threads.

Green/Virtual Threads

Green threads are threads, but not OS threads.

They're threads managed by the application or runtime, that does not involve creating OS threads.

Therefore, they spend less memory as a single OS thread can have multiple virtual threads.

How to create a Green Thread?

Well, you need to replicate what the OS does inside your application/runtime, meaning that you'll need to create an orchestrator (or scheduler) to switch between your virtual threads.

There are two types of schedulers: Preemptive and Cooperative.

Cooperative: It will never block any thread/process. It's the job of the process/thread to give the control back to the scheduler.

Preemptive: It will handle blocking and switching between threads/processes, it's not the job of the application to know when to return the control.

Cooperative are harder to use for the end user, as they'll need to properly handle the stops and switches.

Preemptive is easier to use but harder to create, as it's the job of the scheduler to persist the state of the threads between switches and ensure consistency.

Golang implemented goroutines in it's runtime, by creating a [preemptive scheduler].(https://go.dev/src/runtime/preempt.go)

What about Node.js? We're getting there, hang tight!

Preemptive Scheduler

Ok, let's try to create a preemptive scheduler in Node.js

First, we need a way to add new virtual threads to be called, that's easy!

Let's create a class (I know, JS devs usually hate classes, but I think they're useful sometimes).

class PreemptiveScheduler {
  #virtualThreads;

  constructor() {
    this.#virtualThreads = [];
  }

  addThread(func) {
    this.#virtualThreads.push(func);
  }
}

const longRunningTask = (taskId) => {
  console.time(taskId);
  console.log(`Started running task: ${taskId}`);
  for (let i = 0; i < 100_000_000; i++);
  console.log(`Finished running task: ${taskId}`);
  console.timeEnd(taskId);
};

const scheduler = new PreemptiveScheduler();
scheduler.addThread(() => longRunningTask(1));
scheduler.addThread(() => longRunningTask(2));

Now, we need a way to be able to start the threads, but more important we need a way to be able to stop the running thread, and switch to another one.

And that's where JavaScript (Node.js) cannot help us.

We could add an interval of, let's say, 10ms to access different threads inside the threads array, but, once the function is started, there is no way to stop it.

JavaScript does not provide this functionality.

Meaning that is impossible to create a preemptive scheduler, as this would require being able to stop a function.

Cooperative Scheduler

While we can't create a preemptive scheduler using Node.js or JavaScript, because there is no way to stop a function execution from outside of it, there is one way to cooperatively stop a function, and it is called Generator Functions.

The Node.js Event Loop itself is considered to be a Cooperative Scheduler for MultiTasking, as through callbacks/promises, the user can define when to return the control to the scheduler (Event Loop).

Generator Functions in JavaScript allow us to return the control to the parent function, which can choose whenever it wants to call the next function to unpause the task.

Let's take a look at this code:

class CooperativeScheduler {
  #taskQueue;
  #running;
  #completionResolver;

  constructor() {
    this.#taskQueue = [];
    this.#running = false;
    this.#completionResolver = null;
  }

  addTask(taskGenerator) {
    this.#taskQueue.push(taskGenerator());
  }

  runNextTask() {
    if (!this.#taskQueue.length) {
      this.#running = false;
      if (this.#completionResolver) {
        this.#completionResolver();
      }
      return;
    }

    const currentTask = this.#taskQueue.shift(); // Get first and walk right
    const { done } = currentTask.next(); //Execute next step

    if (!done) {
      // Push next execution to the end of the queue
      this.#taskQueue.push(currentTask);
    }

    setImmediate(() => this.runNextTask());
  }

  start() {
    if (!this.#running && this.#taskQueue.length > 0) {
      this.#running = true;
      this.runNextTask();
    }
  }

  waitForCompletion() {
    if (this.#taskQueue.length === 0 && !this.#running) {
      return Promise.resolve(); // If no tasks are running or pending, resolve immediately
    }

    return new Promise((resolve) => {
      this.#completionResolver = resolve;
    });
  }
}

function* cooperativeFunction(taskId) {
  console.log(`Task ${taskId} started`);
  yield;

  console.log(`Task ${taskId} is processing...`);
  yield;

  console.log(`Task ${taskId} finished!`);
}

// Using the Cooperative Scheduler
(async () => {
  const scheduler = new CooperativeScheduler();
  const times = 10;
  for (let i = 1; i <= times; i++) {
    scheduler.addTask(() => cooperativeFunction(i));
  }
  scheduler.start();
  await scheduler.waitForCompletion();
})();

I start by creating a function *cooperativeFunction(taskId) that is a generator function and it has 2 yield operators.

The yield operator means: to stop the function and return to the caller.

With the CooperativeScheduler class, I've created a mechanism where we can add tasks, start them all and wait for the completion, then, add 10 example tasks.

The scheduler cooperatively pauses and switches between tasks.

This is the main result after running the code:

node cooperative.js

Task 1 started
Task 2 started
Task 3 started
Task 4 started
Task 5 started
Task 6 started
Task 7 started
Task 8 started
Task 9 started
Task 10 started
Task 1 is processing...
Task 2 is processing...
Task 3 is processing...
Task 4 is processing...
Task 5 is processing...
Task 6 is processing...
Task 7 is processing...
Task 8 is processing...
Task 9 is processing...
Task 10 is processing...
Task 1 finished!
Task 2 finished!
Task 3 finished!
Task 4 finished!
Task 5 finished!
Task 6 finished!
Task 7 finished!
Task 8 finished!
Task 9 finished!
Task 10 finished!

And that's a wrap!

Conclusion

Throughout this blog post, we've navigated the intricate landscape of concurrency, shedding light on the distinctions between OS Threads and Virtual/Green Threads. We also delved into the realms of Preemptive and Cooperative Schedulers, exploring their unique characteristics and applications.

This exploration not only highlighted the versatility and challenges of implementing different types of threading models in Node.js but also provided a glimpse into the broader world of concurrent programming.

I hope this post has been enlightening and engaging, offering you valuable insights into the complexities and beauty of threading and concurrency.

Thank you for joining me on this adventure!

Under Pressure: Benchmarking Node.js on a Single-Core EC2

Caio Borghi — Sat, 02 Dec 2023 23:48:51 +0000

Hi!

In this post, I'm going to stress test a Node.js 21.2.0 pure API (no framework!) to see the efficiency of the Event Loop in a limited environment.

I'm using AWS for hosting the servers (EC2) and database (RDS with Postgres).

The main goal is to understand how many requests per second a simple Node API can handle on a single core, then identify the bottleneck and optimize it as much as possible.

Let's dive in!

Infrastructure

AWS RDS running Postgres
EC2 t2.small for the API
EC2 t3.micro for the load tester

Database Setup

The database will consist of a single users table created with the following SQL query:

CREATE TABLE IF NOT EXISTS users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) NOT NULL,
    password VARCHAR(255) NOT NULL
);
TRUNCATE TABLE users;

API Design

The API will have a single POST endpoint that will be used to save a user to the Postgres database. I know, there are a lot of javascript frameworks out there that I could use to make the development easier, but it's possible to use only Node to handle the requests/responses.

To connect to the database, I chose the library pg as it is the most popular one, we'll start with it.

Connection Pooling

One thing that is important when connecting to a database is using a connection pool. Without a connection pool, the API needs to open/close a connection to the database at each request, which is extremely inefficient.

A pool allows the API to reuse connections, as we're planning to send a lot of concurrent requests to our API, it's crucial to have it.

To check your Postgres database's connection limit, run:

SHOW max_connections;

In my case, I'm using an RDS running on a t3.micro database with these specs:

So this is the outcome of the query:

Cool, having 81 as the maximum number of connections to our database, we know what is the upperbound limit we should not surpass.

As the API will run on a single-core processor, it's not a good idea to have a high number of connections on the connection pool, as this would cause a lot of headache to the processor (context switching).

Let's start with 40.

Creating the API

We'll start by starting our project with npm init and creating our index.mjs file. MJS so I can use EcmaScript synthax without doing too much magic/parsing/loading.

The first thing I'll do is add the pg library with npm add pg. I'm using npm but you can use pnpm, yarn or any other node package manager you want.

Then, let's start by creating our connection pool:

import pg from "pg"; // Required because pg lib uses CommonJS 🤢
const { Pool } = pg;

const pool = new Pool({
  host: process.env.POSTGRES_HOST,
  user: process.env.POSTGRES_USER,
  password: process.env.POSTGRES_PASSWORD,
  port: 5432,
  database: process.env.POSTGRES_DATABASE,
  max: 40, // Limit is 81, let's start with 40
  idleTimeoutMillis: 0, // How much time before kicking out an idle client.
  connectionTimeoutMillis: 0, // How much time to disconnect a new client, we don't want to disconnect them for now.
  ssl: false
  /* If you're running on AWS, you'll need to use:
  ssl: {
    rejectUnauthorized: false
  }
  */
});

We're using process.env to access the environment variables, so create a .env file on the root and fill with your postgres informations:

POSTGRES_HOST=
POSTGRES_USER=
POSTGRES_PASSWORD=
POSTGRES_DATABASE=

Then, let's create a function to persist our user on the database.

const createUser = async (email, password) => {
  const queryText =
    "INSERT INTO users(email, password) VALUES($1, $2) RETURNING id";
  const { rows } = await pool.query(queryText, [email, password]);
  return rows[0].id;
};

Finally, let's create a node HTTP server by importing the node:http package and writing a code to handle new requests, parse from string to JSON, query the database and return 201, 400 or 500 in case of any errors, the final file looks like this.

// index.mjs
import http from "node:http";
import pg from "pg";
const { Pool } = pg;

const pool = new Pool({
  host: process.env.POSTGRES_HOST,
  user: process.env.POSTGRES_USER,
  password: process.env.POSTGRES_PASSWORD,
  port: 5432,
  database: process.env.POSTGRES_DATABASE,
  max: 40,
  idleTimeoutMillis: 0,
  connectionTimeoutMillis: 2000,
  ssl: false
  /* If you're running on AWS, you'll need to use:
  ssl: {
    rejectUnauthorized: false
  }
  */
});

const createUser = async (email, password) => {
  const queryText =
    "INSERT INTO users(email, password) VALUES($1, $2) RETURNING id";
  const { rows } = await pool.query(queryText, [email, password]);
  return rows[0].id;
};

const getRequestBody = (req) =>
  new Promise((resolve, reject) => {
    let body = "";
    req.on("data", (chunk) => (body += chunk.toString()));
    req.on("end", () => resolve(body));
    req.on("error", (err) => reject(err));
  });

const sendResponse = (res, statusCode, headers, body) => {
  headers["Content-Length"] = Buffer.byteLength(body).toString();
  res.writeHead(statusCode, headers);
  res.end(body);
};

const server = http.createServer(async (req, res) => {
  const headers = {
    "Content-Type": "application/json",
    Connection: "keep-alive", // Default to keep-alive for persistent connections
    "Cache-Control": "no-store", // No caching for user creation
  };
  if (req.method === "POST" && req.url === "/user") {
    try {
      const body = await getRequestBody(req);
      const { email, password } = JSON.parse(body);
      const userId = await createUser(email, password);

      headers["Location"] = `/user/${userId}`;
      const responseBody = JSON.stringify({ message: "User created" });
      sendResponse(res, 201, headers, responseBody);
    } catch (error) {
      headers["Connection"] = "close";
      const responseBody = JSON.stringify({ error: error.message });
      console.error(error);
      const statusCode = error instanceof SyntaxError ? 400 : 500;
      sendResponse(res, statusCode, headers, responseBody);
    }
  } else {
    headers["Content-Type"] = "text/plain";
    sendResponse(res, 404, headers, "Not Found!");
  }
});

const PORT = process.env.PORT || 3000;

server.listen(PORT, () => {
  console.log(`Server running on http://localhost:${PORT}`);
});

Now, after running npm install, you can run

node --env-file=.env index.mjs

To start the application, you should see this on your terminal:

Congrats, we have built a simple NodeAPI with one endpoint that connects to Postgres through a Connection Pool and inserts a new user to the users table.

Deploying the API to an EC2

First, create an AWS account and go to EC2 > Instances > Launch an Instance.

Then, create an Ubuntu 64-bit (x86) t2.micro instance, allow SSH traffic and allow HTTP traffic from the Internet.

Your summary should look like this:

You'll need to create a key-value-pair.pem file to be able to SSH into it, I won't cover this in this article, there are already plenty of tutorials teaching how to launch and connect to an EC2 instance on the internet, so find them!

Allowing TCP connections on port 3000

After creation, we need to allow TCP traffic for port 3000, this is done on the Security Group config (EC2 > Security Groups > Your Security Group)

At this page, click on "Edit inbound rules", then "Add rule" and fill the form as shown on the image, this will allow us to hit port 3000 of our instance.

Your final Inbound Rules table should look something like this.

Connecting to EC2

Download the .pem file in a folder, then access the EC2 instance and copy the public IPV4 IP, then, run this command on the same folder:

ssh -i <path-to-pen> ubuntu@<public-ipv4-address>

If you see this EC2 welcome page, then you're in 🎉

Installing Node

Let's follow the Node documentation for Debian/Ubuntu-based Linux distros.

Run:

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg

Then:

NODE_MAJOR=21
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | sudo tee /etc/apt/sources.list.d/nodesource.list

Important: Double check that the NODE_MAJOR is 21, as we want to use the latest version of Node <3

sudo apt-get update
sudo apt-get install nodejs -y
node -v

And that's what you should see (it may differ the version as this post gets old)

Nice, now we have a fresh new ubuntu server with node installed, we need to transfer our API code to it and start it.

Deploying API to EC2

We'll use a tool called scp that uses ssh connection to copy file from local to a target location, in our case, the EC2 instance we just created.

Steps:

Delete the node_modules folder from the project.
Go to the parent folder of the root folder of the application.

In my case, the name of the folder is node-api (I know, very creative!)

Now, run:

scp -i <path-to-pem> -r ./node-api ubuntu@<public-ipv4-address>:/home/ubuntu

To transfer the folder node-api to the /home/ubuntu/node-api folder at our EC2 instance.

You should see something similar to this:

Running the API on EC2

Head back to the EC2 server using ssh and run

cd node-api
npm install
NODE_ENV=production node --env-file=.env index.mjs

And boom, the API is running on AWS.

Let's double check that it's working by making a POST request passing email and password to the IP of our API, at the port 3000.

You can use curl (on another terminal), to do this:

curl -X POST -H "Content-Type: application/json" -d {email: user@example.com, password: password} http://<public-ipv4-address>:3000/user

The result should look like this:

I'm using Table Plus to connect to the RDS Postgres database, you could use any Postgres Client.

To ensure that the API is persisting data to the database, let's run this query:

 SELECT COUNT(id) FROM users;

It should return 1.

Nice, it's working!

Stress Test

Now that we have our API working, we need to be able to test how many concurrent requests it can handle with a single core.

There are tons of tools to do this, I'll use Vegeta

You can run the following steps from your local machine, but keep in mind that your network may be the bottleneck, as the stress test requires a lot of packages to be sent at the same time.

I'll use another EC2 instance (a more powerful one, t2x.large) running Ubuntu.

Configuring Vegeta

Follow the docs to install Vegeta on your OS.

Then, create a new folder for load testers on the root folder of the application, it's looking like this:

node_benchmark/
  node-api/
  load-tester/
    vegeta/

Go to the vegeta folder and create a start.sh script with the following content:

#!/bin/bash
if [[ $# -ne 1 ]]; then
    echo 'Wrong arguments, expecting only one (reqs/s)'
    exit 1
fi

TARGET_FILE="targets.txt"
DURATION="30s"  # Duration of the test, e.g., 60s for 60 seconds
RATE=$1    # Number of requests per second
RESULTS_FILE="results_$RATE.bin"
REPORT_FILE="report_$RATE.txt"
ENDPOINT="http://<ipv4-public-address>:3000/user"

# Check if Vegeta is installed
if ! command -v vegeta &> /dev/null
then
    echo "Vegeta could not be found, please install it."
    exit 1
fi

# Create target file with unique email and password for each request
echo "Generating target file for Vegeta..."
> "$TARGET_FILE"  # Clear the file if it already exists

# Assuming body.json exists and contains the correct JSON structure for the POST request
for i in $(seq 1 $RATE); do 
    echo "POST $ENDPOINT" >> "$TARGET_FILE"
    echo "Content-Type: application/json" >> "$TARGET_FILE"
    echo "@body.json" >> "$TARGET_FILE"
    echo "" >> "$TARGET_FILE"
done

echo "Starting Vegeta attack for $DURATION at $RATE requests per second..."
# Run the attack and save the results to a binary file
vegeta attack -rate=$RATE -duration=$DURATION -targets="$TARGET_FILE" > "$RESULTS_FILE"

echo "Load test finished, generating reports..."
# Generate a textual report from the binary results file
vegeta report -type=text "$RESULTS_FILE" > "$REPORT_FILE"
echo "Textual report generated: $REPORT_FILE"

# Generate a JSON report for further analysis
JSON_REPORT="report.json"
vegeta report -type=json "$RESULTS_FILE" > "$JSON_REPORT"
echo "JSON report generated: $JSON_REPORT"

cat $REPORT_FILE

IMPORTANT: Replace <ipv4-public-address> with the IP of your EC2 Node API Server

Now, create a body.json file:

{
  "email": "A1391FDC-2B51-4D96-ADA4-5EEE649A4A75@example.com",
  "password": "password"
}

Now you're ready to start load-testing our api.

This script will:

Run for 30s
Hit the API with concurrent requests/s defined by the first argument of the script
Generate a textual and .json file with infos about the test.

Last, but not least, we need to make the start.sh file executable, we can do this by running:

chmod +x start.sh

Before running each test, I'll clear the users table on Postgres with the following query.

TRUNCATE TABLE users;

This will help us see how many users were created!

1.000 Reqs/s

Alright, let's go to the interesting part, let's see if our single core, 1GB server can handle 1.000 requests per second.

Run

./start.sh 1000

Wait for the completion, here it generates the following output:

Let me break it down for you:

At the rate os 1.000 requests per second, the Node API was able to successfully process all of them, returning the expected success status 201.

In average, each request took 4.254 ms to be returned, with 99% of them returning in less than 25.959 ms.

Metric	Value
Requests per Second	1000.04
Success Rate	100%
p99 Response Time	25.959 ms
Average Response Time	4.254 ms
Slowest Response Time	131.889 ms
Fastest Response Time	2.126 ms
Status Code 201	30000

Let's check our database:

Cool, it worked!

Let's try harder and double the number of requests per second.

2.000 Requests per second

Run

./start.sh 2000

Let's check the output

Awesome, it can handle 2.000 requests/second and still keep a 100% success rate.

Metric	Value
Requests per Second	2000.07
Success Rate	100.00%
p99 Response Time	2.062 s
Average Response Time	136.347 ms
Slowest Response Time	4.067 s
Fastest Response Time	2.164 ms
Status Code 201	60000

A couple things to notice here, while the success rate was still 100%, the p99 jumped from 25.959ms to 2.067s (79x slower than the previous test).

The average response time also jumped from 4.254ms to 136.347 (32.1x slower).

So yeah, doubling the number of requests per second is making our server to suffer A LOT.

Let's try harder and see what happens.

3.000 Requests per second

./start.sh 3000

For 3.000 requests/second our Node.js API started to present problems, being able to process only 52.20%, let's see what happened.

Metric	Value
Requests per Second	2267.72
Success Rate	52.20%
p99 Response Time	30.001 s
Average Response Time	6.146 s
Slowest Response Time	30.156 s
Fastest Response Time	3.018 ms
Status Code 201	36089
Status Code 500	21588
Status Code 0	11465

For 21,588 requests, our API returned status code 500, let's check the API logs:

We can see that our Postgres connection is hitting timeout, the current connectionTimeoutMillis is configured to be 2000 (2s), let's try increasing this to 30000 and see if that improves our load test.

We can do that by changing the line 13 of index.mjs from 2000 to 30000:

connectionTimeoutMillis: 30000

Let's run it again:

./start.sh 3000

And the result?

Metric	Value
Requests per Second	2959.90
Success Rate	97.39%
p99 Response Time	13.375 s
Average Response Time	6.901 s
Slowest Response Time	30.001 s
Fastest Response Time	3.476 ms
Status Code 201	86486
Status Code 0	2318

Nice, by simply increasing the connection timeout for the database we improved the success rate by 45,19%, also, all the 500 errors are now completely gone!

Let's take a look at the remaining errors (status code 0).

Status code 0 usually means that the server reset the connection because it couldn't handle more.

Let's check if it's CPU, Memory or Network.

At the peak of the test, the CPU is only consuming 13%, so it's not CPU.

By running it again with htop I noticed that memory was up only about 70%, so that's also not the problem:

Let's try something different.

File Descriptors

In unix systems, each new connection (socket) is assigned to a File Descriptor. By default, on Ubuntu, the maximum number of open file descriptors is 1024.

You chan check that by running ulimit -n.

Let's try increasing that to 2000 and redo the test to see if we can get rid of these 2% timeout errors.

To do so, I'll follow this tutorial and change to 6000

sudo vi /etc/security/limits.conf

nofile = number of files.
soft = soft limit.
hard = hard limit.

Then reboot the EC2 with sudo reboot now.

After logging in, we can see that the limit changed:

And let's redo the test:

Start the API with

NODE_ENV=production node --env-file=.env index.mjs

And start the load-tester with:

./start.sh 3000

Let's check the results:

Surprisingly, the results are worse!

With a maximum of 2.000 open files, the node API successfully answered only 78.43% of the requests.

This is because by having only one core, adding more open sockets make the processor switch between the files more often than the previous version.

Let's try reducing it to 700 to see if it gets better.

(I'll skip the step on how to do it because it's the same).

And let's see the new output with 700 as maximum open files.

With 700 maximum open files, we hit 83.20% success rate. Let's go back to 1024 and try reducing the connection pool to 20 instead of 40.

If that doesn't work, let's assume 3.000 req/s is slightly higher than the limit and we'll try to find the maximum number of requests/s that a single core node API can handle with 100% success.

With 20 connections for the connection pool, the API was able to process 93.06%, proving that we probably don't need 40.

Let's try with 2.600 reqs/s:

2.600 reqs/s

./start.sh 2600

And that's the result:

Metric	Value
Requests per Second	2600.04
Success Rate	100%
p99 Response Time	8.171 s
Average Response Time	4.573 s
Slowest Response Time	9.234 s
Fastest Response Time	5.244 ms
Status Code 201	77999

That's a wrap!

Conclusion

This experiment demonstrates the capabilities of a pure Node.js API on a single-core server.

With a pure Node.js 21.2.0 API, using a single core with 1GB of RAM + connection pool with a maximum 20 connections, we were able to achieve 2.600 requests/s without failures.

By fine-tuning parameters like connection pool size and file descriptor limits, we can significantly impact performance.

What's the highest load your Node.js server has handled? Share your experiences!

Profundezas do Node.js: Explorando I/O Assíncrono

Caio Borghi — Sat, 02 Sep 2023 13:21:27 +0000

Introdução
Como o Node trata o Código Assíncrono
Operações Assíncronas: O que São?
- Operação Assíncrona Bloqueante vs Não Bloqueante
Experimentos com Funções Bloqueantes
Experimentos com Funções Não-Bloqueantes
Operações Assíncronas Não Bloqueantes e SO
- Entendendo File Descriptors
  - O que é FD?
  - FD e I/O não bloqueante
- Monitorando FDs com syscalls
  - Entendendo o select
  - Epoll
  - io_uring
Conclusão

Introdução

Recentemente tenho estudado sobre execução de código assíncrono no Node.js.

Acabei aprendendo (e escrevendo) bastante coisa, desde um artigo sobre como o Event Loop funciona até uma thread no Twitter explicando quem espera a requisição http terminar.

Se você quiser, pode também acessar o mapa mental que criei antes de escrever esse post, clicando aqui

Agora, vamos ao que interessa!

Como o Node trata o Código Assíncrono

No node:

Todo código JavaScript é executado na thread principal.
A biblioteca libuv é encarregada de lidar com operações de I/O (In/Out), ou seja, operações assíncronas.
Por padrão, o libuv disponibiliza 4 worker threads para o Node.js
- Essas threads só serão utilizadas quando operações assíncronas bloqueantes forem realizadas, nesse caso, bloquearão uma das threads do libuv (que são threads do Sistema Operacional) ao invés da thread principal (de execução do Node).
Existem operações bloqueantes e não bloqueantes, a maioria das operações assíncronas atuais são não bloqueantes.

Operações Assíncronas: O que São?

Geralmente, existe uma confusão quando se trata de operações assíncronas.

Muitos acreditam que significa que algo ocorre em segundo plano, em paralelo, ao mesmo tempo ou em uma outra thread.

Na verdade, uma operação assíncrona é uma operação que não retornará agora, mas sim depois.

Elas dependem de uma comunicação com agentes externos e, esses agentes, podem não ter uma resposta imediata para sua solicitação.

Estamos falando de operações de I/O (entrada/saída).

Exemplos:

Leitura de arquivo: dados saem do disco e entram na aplicação.
Escrita em um arquivo: dados saem da aplicação e entram no disco.
Operações de Rede
- Requisições HTTP, por exemplo.
- A aplicação envia uma requisição http para algum servidor e recebe os dados.

Operação Assíncrona Bloqueante vs Não Bloqueante

No mundo moderno, ~~as pessoas não se falam~~ a maioria das operações assíncronas não bloqueiam.

Mas peraí, isso quer dizer que:

a libuv disponibiliza 4 threads (por padrão).
elas "cuidam" das operações de I/O bloqueantes.
a grande maioria das operações são não-bloqueantes.

Parece meio inútil né?

Com esse questionamento em mente, resolvi fazer alguns experimentos.

Experimentos com Funções Bloqueantes

Primeiro, testei uma função assíncrona de uso intenso de CPU, uma das raras funções assíncronas bloqueantes do Node.

O código utilizado foi o seguinte:

// index.js
// index.js
import { pbkdf2 } from "crypto";

const TEN_MILLIONS = 1e7;

// Função assíncrona de uso intenso de CPU
// Objetivo: Bloquear uma worker thread
// Objetivo original: Gerar uma palavra-chave
// O terceiro parâmetro é o número de iterações
// Nesse exemplo, estamos passando 10 milhões
function runSlowCryptoFunction(callback) {
  pbkdf2("secret", "salt", TEN_MILLIONS, 64, "sha512", callback);
}

// Aqui queremos saber quantas workers threads a libuv vai usar
console.log(`Thread pool size is ${process.env.UV_THREADPOOL_SIZE}`);

const runAsyncBlockingOperations = () => {
  const startDate = new Date();
  const runAsyncBlockingOperation = (runIndex) => {
    runSlowCryptoFunction(() => {
      const ms = new Date() - startDate;
      console.log(`Finished run ${runIndex} in ${ms/1000}s`);
    });
  }
  runAsyncBlockingOperation(1);
  runAsyncBlockingOperation(2);
};

runAsyncBlockingOperations();

Para validar o funcionamento, eu rodei o comando:

UV_THREADPOOL_SIZE=1 node index.js

IMPORTANTE:

UV_THREADPOOL_SIZE: É uma variável de ambiente que determina quantas worker threads da libuv o Node vai iniciar.

O resultado foi:

Thread pool size is 1
Finished run 1 in 3.063s
Finished run 2 in 6.094s

Ou seja, com 1 única thread, cada execução levou ~3 segundos e elas ocorreram de forma sequencial. Uma após a outra.

Agora, resolvi fazer o seguinte teste:

UV_THREADPOOL_SIZE=2 node index.js

E o resultado foi o seguinte:

Thread pool size is 2
Finished run 2 in 3.225s
Finished run 1 in 3.243s

Com isso, está provado que as Worker Threads da LIBUV, no Node.js lidam com operações assíncronas bloqueantes.

Mas e as não bloqueantes? Se ninguém espera por elas, como elas funcionam?

Eu resolvi escrever uma outra função para fazer o teste.

Experimentos com Funções Não-Bloqueantes

A função fetch (nativa do Node) realiza uma operação assíncrona de rede e ela é não-bloqueante.

Com o seguinte código, refiz o teste do primeiro experimento:

//non-blocking.js
// Aqui queremos saber quantas workers threads a libuv vai usar
console.log(`Thread pool size is ${process.env.UV_THREADPOOL_SIZE}`);

const startDate = new Date();
fetch("https://www.google.com").then(() => {
  const ms = new Date() - startDate;
  console.log(`Fetch 1 retornou em ${ms / 1000}s`);
});

fetch("https://www.google.com").then(() => {
  const ms = new Date() - startDate;
  console.log(`Fetch 2 retornou em ${ms / 1000}s`);
});

E executei o script com o seguinte comando:

UV_THREADPOOL_SIZE=1 node non-blocking.js

O resultado foi o seguinte:

Thread pool size is 1
Fetch 1 retornou em 0.391s
Fetch 2 retornou em 0.396s

Então, resolvi testar com duas threads, para ver se mudava algo:

UV_THREADPOOL_SIZE=2 node non-blocking.js

E então:

Thread pool size is 2
Fetch 2 retornou em 0.402s
Fetch 1 retornou em 0.407s

Com isso, pude observar que:

Ter mais threads rodando na LIBUV não ajuda na execução de operações assíncronas não bloqueantes.

Mas então, voltei a me questionar, se nenhuma thread da libuv fica "esperando" a requisição voltar, como é que isso funciona?

Meu amigo, foi aí que eu caí num gigantesco buraco de pesquisa e conhecimentos sobre o funcionamento de:

Operações Assíncronas Não Bloqueantes e SO

O Sistema Operacional evoluiu bastante com o passar dos anos para conseguir lidar com operações de I/O de forma não bloqueante, isso é feito através de syscalls, são elas:

select/poll: Estas são as formas tradicionais de lidar com I/O não bloqueante e são geralmente consideradas menos eficientes.
IOCP: Usado no Windows para operações assíncronas. kqueue: Um método para MacOS e BSD.
epoll: Eficiente e utilizado no Linux. Ao contrário de select, ele não é limitado pelo número de FDs.
io_uring: Uma evolução do epoll, trazendo melhorias de desempenho e uma abordagem baseada em filas.

Para entendermos melhor, vamos precisar mergulhar nos detalhes das operações de I/O não bloqueante.

Entendendo File Descriptors

Para conseguir explicar I/O não bloqueante, preciso explicar rapidamente o conceito de File Descriptors (FDs).

O que é FD?

É um índice numérico de uma tabela mantida pelo kernel, onde cada registro possui:

Tipo do recurso (como arquivo, soquete, dispositivo).
Posição atual do ponteiro do arquivo.
Permissões e flags, definindo modos como leitura ou escrita.
Referência à estrutura de dados do recurso no kernel.

Eles são fundamentais para o gerenciamento de I/O.

FD e I/O não bloqueante

Ao iniciar operação de I/O não bloqueante, o Linux atrela um FD a ela sem interromper (bloquear) a execução do processo.

Por exemplo:

Vamos imaginar que você quer ler o conteúdo de um arquivo muito grande.

Abordagem bloqueante:

Processo chama a função ler arquivo
Processo aguarda o SO ler o conteúdo do arquivo
- Enquanto o SO não terminar, o processo está bloqueado

Abordagem não bloqueante:

Processo solicita leitura assíncrona.
O SO começa a ler o conteúdo e retorna FD para o processo.
Processo não está travado e pode fazer outras coisas.
De tempos em tempos, o processo chama uma syscall para saber se a leitura acabou.

Quem define o modo como a leitura será feita é o processo, através da função fcntl com a flag O_NONBLOCK, mas isso é secundário no momento.

Monitorando FDs com syscalls

Para observar múltiplos FDs de maneira eficiente, os SOs contam com algumas syscalls:

Entendendo o select:

Recebe uma lista de FDs.
Bloqueia o processo até que um ou mais FDs estejam prontos para a operação especificada (leitura, escrita, exceção).
Após o retorno da syscall, o programa pode iterar sobre os FDs para identificar os que estão prontos para I/O.
Utiliza algoritmo de busca que é O(n).
- Ineficiente, lento, cansado com muitos FDs

Epoll

Foi uma evolução do select, utiliza uma árvore auto-balanceada para armazenar os FDs, fazendo com que o tempo de acesso seja praticamente constance, O(1).

Chique demais!

Como funciona:

Cria-se uma instância do epoll através de epoll_create.
Associa os FDs a essa instância com epoll_ctl.
Usa epoll_wait para aguardar atividade em algum dos FDs.
Possui parâmetro de timeout.
- Extremamento importante e muito bem utilizado pelo Event Loop da libuv!

Io_uring

Esse cara aqui veio para chutar o pau da barraca.

Enquanto o epoll evoluiu (e muito!) o desempenho de busca e apreensão dos FDs, o io_uring veio para repensar toda a natureza das operações de I/O.

E assim, depois de entender como ele funciona, fiquei me perguntando como ninguém pensou nisso antes!!!

Recapitulando:

select: Recebe uma lista de FDs, armazena-os sequencialmente (como um array) e verifica 1 a 1 (complexidade O(n)) para ver quem teve alteração ou atividade.
epoll: Recebe uma lista de FDs, armazena-os utilizando uma árvore auto-balanceada, não verifica 1 a 1, é mais eficiente, e faz o mesmo que o select só que com complexidade O(1)

Historicamente, o processo ficava encarregado de iterar sobre os FDs retornados para saber quem terminou ou não.

io_uring: Como é que é? Retornar uma lista? Fazer polling? Cês são burros? Já ouviram falar de filas?

Ele funciona utilizando duas filas principais, na forma de anéis (rings, daí o nome io-ring).

1 para submeter tarefas
1 para tarefas concluídas

Simples né?

O processo, ao iniciar uma operação de I/O, enfileira a operação utilizando a estrutura io_uring.

Aí, ao invés de chamar select ou epoll e, com os FDs retornados ficar iterando sobre cada um deles, o processo pode optar por ser notificado quando alguma operação de I/O acabar.

Polling? Não. Filas!

Conclusão

Com isso, agora eu sei exatamente qual é o caminho que o Node percorre para realizar uma operação assíncrona.

Se é bloqueante:

Executa a operação assíncrona utilizando a libuv
Adiciona a uma worker thread da libuv
A worker thread fica bloqueada, esperando a operação terminar.
Ao terminar, a thread se encarrega de colocar o resultado no Event Loop na fila de MacroTasks
O callback é executado na thread principal

Se não é bloqueante:

Executa a operação assíncrona utilizando a libuv
Libuv executa uma syscall de I/O não bloqueante
Faz polling com os FDs até que se resolvam (epoll)
A partir da versão 20.3.0 utiliza io_uring
- Abordagem de filas de submissão/operações completadas
Ao receber evento de operação completada
- libuv se encarrega de executar o callback na thread principal

O Futuro do Trabalho: Explorando a Cultura Async First.

Caio Borghi — Sun, 27 Aug 2023 18:16:46 +0000

Já está provado que o trabalho 100% remoto aumenta a qualidade de vida dos trabalhadores.

Seja pela economia do não-deslocamento, a maior proximidade com a família, maior customização do espaço ou simplesmente pelo maior conforto de, assim como Cid Moreira, poder trabalhar de bermuda.

Porém, junto com a popularização do trabalho remoto, veio o aumento das reuniões.

Reuniões diárias, semanais, de planejamento, estimativas, revisões, debates, atualizações, team building, 1-1, all hands e de mais muitos outros tipos.

Reuniões que, a meu ver, tentam replicar o modelo de trabalho existente há décadas no escritório, só que dessa vez, no mundo digital.

Nesse post, pretendo explicar um pouco sobre a cultura Async First, que quebra um pouco o paradigma clássico do modelo de trabalho e propõe uma abordagem mais moderna que, além de fazer muito mais sentido para o mundo digital, estimula a independência do trabalhador e fornece mais liberdade/flexibilidade para o seu dia a dia.

O que é Async First?

Async First, ou "assíncrono em primeiro lugar", é uma abordagem que prioriza a comunicação assíncrona ao invés de interações em tempo real.

Mas o que é comunicação assíncrona?

Vou começar com exemplos:

Comunicação síncrona

Conversa cara-a-cara
Vídeo-chamadas
Chamadas telefônicas
Bate-papo/Chat de mensagens instantâneas

Comunicação assíncrona

Emails
Gravações de tela
Vídeos explicativos
Mensagens de áudio gravadas
Comentários em documentos compartilhados (como Google Docs ou Microsoft Word online).
Postagens em fóruns ou painéis de discussão.
Tarefas ou comentários em sistemas de gerenciamento de projetos (como Trello, Asana, etc.).
SMS ou mensagens de texto (quando não há expectativa de resposta imediata).

Isso significa que, ao invés de todos estarem online ao mesmo tempo para uma reunião ou discussão, as informações são compartilhadas de uma forma que permite às pessoas acessá-las e respondê-las em seu próprio tempo.

Com essa abordagem, não existe a necessidade de duas pessoas estarem online no mesmo tempo. Com ela, seria possível realizar o seu trabalho às 08 da manhã, às 02 da tarde ou em qualquer outro horário que você preferir.

Quando o assunto é desenvolvimento de software, a grande maioria das tarefas poderia ser feita de maneira assíncrona.

Será que isso funciona? Bem, essa é a lista de empresas que adotaram a cultura Async First:

GitLab
Atlassian (Dona do Jira, Trello, Confluence e Bitbucket)
Gumroad
Zapier
Toggl

Benefícios do Async First

Maior Flexibilidade: Cada membro da equipe tem a liberdade de organizar sua agenda e decidir quando será mais produtivo, em vez de estar preso a um horário fixo de reuniões ou interações.
Foco Profundo: Sem a necessidade constante de interrupções para reuniões em tempo real, os trabalhadores podem mergulhar profundamente em suas tarefas e aumentar sua produtividade.
Inclusividade Global: Equipes distribuídas ao redor do mundo não precisam se preocupar com fusos horários conflitantes, tornando mais fácil e incentivando as contratações globais.
Redução da Fatiga Digital: Reuniões são cansativas, menos reuniões = menos cansaço.
Documentação Eficiente: A comunicação assíncrona muitas vezes é documentada, o que é muito útil, pois todos podem revisitar informações sempre que necessário. Atualmente, se ninguém anota nada durante uma reunião, a equipe depende 100% da memória humana para guardar informações.

Como garantir uma cultura eficiente

É importante utilizar boas ferramentas de documentação, manter uma wiki atualizada e organizada para facilitar o acesso de toda informação relevante à cultura da empresa e projetos.

Além disso, é crucial estabelecer um equilíbrio, pois algumas discussões e decisões podem se beneficiar de interações em tempo real.

Async First, ou "Comunicação assíncrona por padrão" é diferente de Sync Never ou "Nunca ter reunião".

Uma equipe ainda pode fazer reuniões, mas incentivando a comunicação assíncrona primeiro, a frequência, duração e necessidade de reuniões diminuirão ao ponto de não serem mais um incômodo.

Em resumo, enquanto o trabalho remoto se tornou um novo padrão para muitas empresas, a abordagem Async First traz uma poderosa evolução!

Propõe um caminho que valoriza a autonomia do trabalhador e reconhece o potencial do mundo digital e globalizado.

O mundo mudou muito nos últimos 30 anos, mas os modelos de trabalho não acompanharam.

Eu sou à favor da cultura Async First, espero que se torne cada vez mais popular.

E você, o que acha? Fique à vontade para deixar sua opinião, ponto de vista ou crítica nos comentários, acredito que promover esse debate é crucial para repensarmos os modelos de trabalho atuais.

Valeu!

JavaScript Event Loop: Breaking Down the Mystery

Caio Borghi — Thu, 03 Aug 2023 16:40:38 +0000

What happens when the following code is executed in Node.js?

setTimeout(() => console.log(1), 10)
Promise.resolve().then(() => console.log(2))
console.log(3)

If your answer was different from:

3
2
1

Perhaps you don't fully understand the execution order of JavaScript and the operation of the Event Loop.

No worries, I'll try to explain.

First of all, if you have doubts about what is:

JavaScript
ECMAScript
JavaScript Runtime

I recommend that you read the glossary before continuing.

Now, let's go, I will explain what happens at each stage of the execution of this JavaScript code.

Main Thread

Node interprets the JavaScript file from top to bottom, line by line, in a single thread.

Running setTimeout()

The main thread will interpret the first instruction, add it to the Call Stack, where it will be executed and removed from the Call Stack.

The setTimeout instruction is used to schedule the execution of a function after certain milliseconds.

This function is part of the libuv library, which Node uses to create a Timer without blocking the main thread.

After starting the Timer, the main thread will remove the instruction from the Call Stack.

At the end of the interval, the timer will add the callback of the setTimeout function to the macro-task queue.

Running Promise.resolve().then()

While the Timer of the libuv library waits for the 10ms, the Main thread will interpret the next line of the file.

The instruction this time is

Promise.resolve().then(() => console.log(2))

The main thread will execute the function Promise.resolve().then()

Promise is an object that represents a completion or failure of an asynchronous operation.

By calling the resolve() function without any parameter, we are declaring Promise that does not return any value, but that's okay.

For now, we are more interested in the behavior of the .then function of a Promise.

By passing () => console.log(2) as callback for our Promise, we are telling Node to execute this code as soon as the Promise is successfully finished.

In other words, we are saying that, as soon as the resolve() method of the Promise is executed, Node should execute our console.log(2) instruction.

But, that's not exactly how it works.

Every Promise callback is sent instantly to a special queue called Micro Tasks Queue.

Recapping

This is the current state of the script execution:

Everything that happened so far, surely, took less than 10 milliseconds, which is why the Timer has not yet added the instruction of console.log(1) to the Macro Tasks Queue.

But, by using libuv, the Main thread can continue working normally, in a non-blocking manner.

Ok, you might be wondering:

Event Loop

Throughout this process, with each interpretation of a new line from the file, the Event Loop performed a very important, albeit repetitive function.

Check if the Call Stack was empty.

As you can see, the answer was always: NO!

At no time during the execution of this script was the Call Stack empty, so our friend Event Loop will keep waiting.

Emptying the Call Stack

Now, the Main Thread interprets the last instruction of the file.

This is a simple instruction, which displays a value on the console, its result is:

And, for the first time, the Call Stack is empty!

Event Loop

Now, the most awaited moment for the Event Loop, the moment when it has the power to act!

It will only validate the other queues when the Call Stack is empty!

At each loop, it will:

Process all tasks in the Micro Tasks queue
- Adding them to the Call Stack
Process 1 task from the Macro Tasks queue
- Adding it to the Call Stack
Wait for the Call Stack to empty
Repeat

The Main Thread executes every instruction in the main context.

Now, continuing the execution of the example code:

Micro Tasks

When the Call Stack becomes empty, it means that the Main Thread is not executing anything.

Then, the Event Loop consumes all tasks from the Micro Tasks Queue and adds them to the Call Stack.

Next, the Main Thread consumes the instruction from the Call Stack and executes it.

console.log(2) // Writes 2 to the console

Now, the Call Stack becomes empty again.

Then, the Event Loop looks for more tasks in the Micro Tasks queue.

As it is empty, it finishes its work in the Micro Tasks Queue and starts consuming the Macro Tasks Queue.

Macro Tasks

Now, suppose that the 10-millisecond interval has passed and the Timer has inserted the console.log(1) function into the Macro Tasks queue, the Event Loop will transfer 1 instruction from the Macro Tasks Queue to the Call Stack.

Then, the Main Thread consumes the last instruction from the Call Stack and executes it.

console.log(1) // Writes 1 to the console

Important point: If there were still instructions in the Micro Tasks queue, these would be processed. But, as everything is empty, the program execution is heading towards the end.

That's why the code:

setTimeout(() => console.log(1), 10)
Promise.resolve().then(() => console.log(2))
console.log(3)

Will result in:

3
2
1

We've reached the end - Arlindo Cruz

Now you understand what happens behind the scenes of JavaScript. The Event Loop manages the queues of micro and macro tasks and, with that, ensures that asynchronous instructions are executed harmoniously in the context of the main thread.

Understanding how it works helps us write more efficient codes and better predict the behavior of our applications.

Next time you're writing JavaScript code, I hope you remember everything that happens behind the scenes of the Event Loop.

See you later!

Glossary

JavaScript

It's a high-level, dynamic, interpreted programming language that supports multiple programming paradigms (functional, imperative, object-oriented).

It's a "medium" of conversation between something you want to do and what the computer executes.

ECMAScript

It's a set of rules that defines how JavaScript should work, it defines the language standards (syntax, data types, control structures, and operators), and JavaScript is the implementation of these standards.

If you want to understand better, read this article

JavaScript Runtime

It's the engine that executes JavaScript code.

When writing JavaScript code, you write instructions (which follow the rules defined by ECMAScript), but to execute these instructions, you need a Runtime.

It's as if JavaScript were a recipe and the Runtime was a cook who executes the recipe.

Node, V8 and SpiderMonkey are the most well-known JavaScript runtimes in the world.

A Magia do Event Loop

Caio Borghi — Wed, 02 Aug 2023 00:32:31 +0000

O que acontece quando o seguinte código é executado no Node.js?

setTimeout(() => console.log(1), 10)
Promise.resolve().then(() => console.log(2))
console.log(3)

Se a sua resposta foi diferente de:

3
2
1

Talvez você não entenda muito bem a ordem de execução do JavaScript e o funcionamento do Event Loop.

Sem problemas, vou tentar explicar.

Antes de tudo, se você tem dúvidas sobre o que é:

JavaScript
ECMAScript
Runtime de JavaScript

Eu recomendo que você leia o glossário antes de continuar.

Agora, vamos lá, vou explicar o que acontece em cada etapa da execução desse código JavaScript.

Main Thread

O Node interpreta o arquivo JavaScript de cima para baixo, linha por linha, em uma única thread.

Executando o setTimeout()

A main thread interpretará a primeira instrução, adicionará na Call Stack, onde será executada e removida da Call Stack.

A instrução setTimeout serve para agendar a execução de uma função após determinados millisegundos.

Essa função faz parte da biblioteca libuv, que o Node utiliza para criar um Timer sem bloquear a thread principal.

Após iniciar o Timer, a thread principal removerá a instrução da Call Stack.

Ao final do intervalo, o timer vai adicionar o callback da função setTimeout na fila de macro-tarefas.

Executando Promise.resolve().then()

Enquanto o Timer da biblioteca libuv espera os 10ms, a Main thread interpretará a próxima linha do arquivo.

A instrução da vez é a

Promise.resolve().then(() => console.log(2))

A thread principal vai executar a função Promise.resolve().then()

Promise é um objeto que representa uma conclusão ou falha de uma operação assíncrona.

Ao chamar a função resolve() sem nenhum parâmetro, estamos declarando Promise que não retorna valor algum, mas tudo bem.

Por ora, estamos mais interessados no comportamento da função .then de uma Promise.

Ao passar () => console.log(2) como calback para nossa Promise, estamos dizendo para o Node executar este código assim que a Promise for finalizada com sucesso.

Ou seja, estamos dizendo que, assim que o método resolve() da Promise for executado, o Node deverá executar nossa instrução console.log(2).

Mas, não é bem assim que funciona.

Todo callback de Promise é enviado instantaneamente para uma fila especial chamada Micro Tasks Queue.

Recapitulando

Esse é o estado atual da execução do script:

Tudo que aconteceu até agora, com certeza, levou menos de 10 millissegundos, por isso que o Timer ainda não adicionou a instrução de console.log(1) na Macro Tasks Queue.

Mas, por utilizar a libuv, a Main thread pode continuar trabalhando normalmente, de maneira não-bloqueante.

Ok, você pode estar se perguntando:

Event Loop

Durante todo esse processo, a cada interpretação de nova linha do arquivo, o Event Loop realizou uma função muito importante, embora repetitiva.

Verificar se a Call Stack estava vazia.

Como você pode observar, a resposta foi sempre: NÃO!

Em nenhum momento durante a execução desse script a Call Stack ficou vazia, então, nosso amigo Event Loop continuará esperando.

Esvaziando a Call Stack

Agora, a Main Thread interpreta a última instrução do arquivo.

Essa é uma instrução simples, que exibe um valor no console, seu resultado é:

E, pela primeira vez, a Call Stack fica vazia!

Event Loop

Agora sim, o momento mais esperado pelo Event Loop, o momento que ele tem o poder de agir!

Ele só vai validar as outras filas quando a Call Stack estiver vazia!

A cada loop, ele vai:

Processar todas as tarefas da fila de Micro Tasks
- Adicionando-as à Call Stack
Processar 1 tarefa da fila de Macro Tasks
- Adicionando-a à Call Stack
Esperar a Call Stack esvaziar
Repetir

A Main Thread executa toda instrução no contexto principal.

Agora, continuando a execução do código de exemplo:

Micro Tasks

Quando a Call Stack fica vazia, significa que a Main Thread não está executando nada.

Então, o Event Loop consome todas as tarefa da Micro Tasks Queue e adiciona à Call Stack

Em seguida, a Main Thread consome a instrução da Call Stack e a executa.

console.log(2) // Escreve 2 no console

Agora, a Call Stack volta a ficar vazia.

Então, o Event Loop busca por mais tarefas na fila de Micro Tasks.

Como está vazia, ele finaliza o seu trabalho na Micro Tasks Queue e vai começar a consumir a Macro Tasks Queue.

Macro Tasks

Agora, supondo que o intervalo de 10 millisegundos já tenha passado e o Timer tenha inserido a função de console.log(1) na fila de Macro Tasks, o Event Loop transferirá 1 instrução da Macro Tasks Queue para a Call Stack.

Então, a Main Thread consome a última instrução da Call Stack e a executa.

console.log(1) // Escreve 1 no console

Ponto importante: Se ainda houvesse instruções na fila de Micro Tasks, estas seriam processadas. Mas, como tudo está vazio, a execução do programa caminha para o fim.

É por isso que o código:

setTimeout(() => console.log(1), 10)
Promise.resolve().then(() => console.log(2))
console.log(3)

Resultará em:

3
2
1

Chegamos ao fim - Arlindo Cruz

Agora você entende o que acontece nos bastidores do JavaScript. O Event Loop gerencia as filas de micro e macro tarefas e, com isso, garante que instruções assíncronas sejam executadas com harmonia no contexto da thread principal.

Entender seu funcionamento nos ajuda a escrever códigos mais eficientes e a prever melhor o comportamento de nossas aplicações.

Da próxima vez que for escrever código em JavaScript, espero que se lembre de tudo que acontece nos bastidores do Event Loop.

Até mais!

Glossário

JavaScript

É uma linguagem de programação de alto nível, dinâmica, interpretada, que suporta múltiplos paradigmas de programação (funcional, imperativo, orientado a objetos).

É um "meio" de conversa entre algo que você quer fazer e que o computador executa.

ECMAScript

É um conjunto de regras que define como o JavaScript deve funcionar, ela define os padrões da linguagem (sintaxe, tipos de dados, estruturas de controle e operadores), e o JavaScript é a implementação desses padrões.

Se quiser entender melhor, leia esse artigo

Runtime JavaScript

É o motor que executa o código JavaScript.

Ao escrever código JavaScript, você escreve instruções (que seguem as regras definidas pelo ECMAScript), mas para executar essas instruções, você precisa de um Runtime.

É como se o JavaScript fosse uma receita e o Runtime fosse um cozinheiro que executa a receita.

Node, V8 e SpiderMonkey são os runtimes JavaScript mais conhecidos do mundo.

DEV Community: Caio Borghi

React Query - why does it matter?

useEffect hell

Code difference

bad

good

Why bad?

caching

refetching

Go 1.24 uses Swiss Tables, what are they?

Introduction

The Old Map

Chaining

Practical Example

Problem

Swiss Table

Linear Probing

Steroids (SSE3)

Metadata

1.23.4 vs 1.24

Results

Problem

Elastic Hashing

What is it?

Magic

Conclusion

References

Benchmarking DeepSeek R1 on a Developer’s MacBook

Table of Contents

Introduction

What's the point?

The Prompt

What is Time?

The Answer

How were the metrics gathered?

Sequence Diagram

Process Monitor

GPU Monitor

Requests Metrics

Hardware Specification

Tools

Ollama

Golang

Python

Benchmark Results

Waiting Time (TTFB)

TTFB Table Data

Velocity

Comparing different token/s speed

55 tokens per second

30.7 tokens per second

9.1 tokens per second

100 tokens/s

Acceptable Thresholds

Maximum Acceptable Wait Time

Minimum Acceptable Response Speed

Throughput + Wait Time

Throughput + Wait Time Table Data

Duration

Requests Average Duration

Combined Metrics (Tokens/s x Duration x Wait Time)

GPU Usage

RAM/CPU/Threads Usage

Web Server Process

DeepSeek Process

Summarized Results

How many tokens/s can I get running DeepSeek R1 Gwen 7B locally with ollama?

How many parallel requests can I serve with reasonable throughput?

What is a reasonable throughput?

How does the number of concurrent requests impact the performance?

Conclusion

The Evolution of React State Management: From Local to Async

Table of Contents

Introduction

Local State

Global State

What is it?

How to use it?

The main way

The simple way