DEV Community: Lucas Weis Polesello

The Zalgo Effect and Resource Leakage - A Case

Lucas Weis Polesello — Sun, 17 Mar 2024 23:29:58 +0000

First of all, what's the `Zalgo Effect`?

Zalgo Effect is a term used to describe unexpected outcomes of mixing sync and async JavaScript code - meaning if you mix these two approaches ~SOMETHING~ weird will happen.
It's one of those things you kinda don't understand until you see it in real production systems.

Well, in the internet and Pop culture - it's more than that. Think of it like a monster.

So what it has to do with Resource Leakage?

Occasionally, our SRE team received a couple PagerDuty alerts claiming our services were restarting and not able to work properly due to Error: No channels left to allocate - ie RabbitMQ connections were maxing out in channel allocation. (For RabbitMQ reference into Channels and Connections)

(Ofc the screenshot is just an old print)

It was clear some code was leaking channel creations. No one knew what could potentially be - but God I listened somewhere about this Zalgo Effect

This "somewhere" is called Node.js Design Patterns: Design and Implement Production-grade Node.js Applications Using Proven Patterns and Techniques - Packet book.

How was I so sure this "Zalgo" was the culprit?

The service throwing that error was only responsible to fan out a couple messages for a lot of other services - so it was easy as creating a our internal Queue object and running N promises concurrently to publish some message - as the first thing the script run.
The RabbitMQ Management UI showed me that we created one channel for each promise - roughly.

But why it only happened in some scenarios?

That's where the Zalgo Effect pops in.

The PubSub code was built back in ~2015 - Node 4 - where the callback style was the go-to. Our Engineers created the abstraction Queue which dealt with almost 50% of our Event-Driven Architecture by itself. It was a very naive implementation - pointing to a single RabbitMQ Node.

Our legacy code was 100% usage of async.series - so when publishing messages it would be always sequentially. Later then parallel processing was added, issues started to appear more frequently. It was always just an silent issue.

So the code assumed the following:

Assert the exchange, queues and necessary resources - using one channel - which we could call consumeChannel.
1. The consume channel is created whenever the connection is made - and it is always created.
Our confirmChannel - ie the channel we used to publish events was lazily created - only being created when we need to publish some message via that connection.
1. This code mixed async and sync code heavily

Imagine the following:
We call assertConfirmChannel - which calls the AmqpLibConfirmChannelSingleton and then it creates the channel. If the Singleton, still has no inner-instance to refer, it calls a promise to create the confirmChannel and attaches the error listeners. Code example below:

What happens if two concurrent promises reaches the `AmqpConfirmChannelSingleton.getInstance` at the same time - in the same event-loop tick?

Code would reach this line from amqpLibChannelDecorator.createConfirmChannel - N times where the Promise is not resolved yet.
It means, we would have N promises created - mistakenly creating useless channels and de-referencing them.

The NodeJS GC won't kill this - since the channel is just an abstraction and still has "reference" inside the connection.

This is where the code was leaking channels.

Fixing the problem

The hotfix

Due to production hours - our hot-fix was to simply await the first promise and then fan out the other promises.
The real fix - below - was shipped with the PubSub refactoring.

How did we fix this in the long term?

If you want a real solution, here's what the V2 looked like - the idea is to create Promises and assign variables with them, instead of doing await on it. Example as below:

Notice how this.connection has an unawaited promise.

This easily fixes the problem - by setting a variable as promise and checking its existence.

A more robust style example - where you actually need to initialize a couple of resources, you could do something like below - which is the exact same idea.

Create a function to execute the entire Promise.
Set up some reference to it
If requested the same, just use the same Promise.

Ok - but why it fixes the problem?

NodeJS runs upon the famous Event-loop.

TL;DR: It is an ever-running while(true) with some well-defined and ordered steps from a queue of functions to execute.

Basically, what happens in this example is - too much synchronous code executing without ever reaching the Poll phase - where the IO callbacks are executed.

Thanks for reading it and leave a comment!

Redis is more than a Cache #1 - Delaying Jobs

Lucas Weis Polesello — Fri, 15 Mar 2024 23:18:06 +0000

My current company - Luma Health Inc - has an Event-Driven Architecture where all of our backend systems interact via async messaging/jobs. Thus our backbone is sustained by an AMQP broker - RabbitMQ - which routes the jobs to interested services.

Since our jobs are very critical - we cannot support failures AND should design to make the system more resilient - because well...we don't want a patient not being notified of their appointment, appointments not being created when they should, patients showing off into facilities where they were never notified the patient had something scheduled.

Besides the infra and product reliability - some use cases could need postponing - maybe reaching out to an external system who's offline/or not responding. Maybe some error which needs a retry - who knows?

The fact is, delaying/retrying its a very frequent requirement into Event Driven Architectures. With this a service responsible for doing it was created - and it worked fine.

But - as the company sold bigger contracts and grew up in scale - this system was almost stressed out and not reliable.

The Unreliable Design

Before giving the symptoms, let's talk about the organism itself - the service old design.

The design was really straightforward - if our service handlers asked for a postpone OR we failed to send the message to RabbitMQ - we just insert the JSON object from the Job into a Redis Sorted Set and using the Score as the timestamp which it was meant to be retried/published again.

To publish back into RabbitMQ the postponed messages, a job would be triggered each 5 seconds - doing the following:

Read from a set key containing all the existing sorted set keys - basically the queue name
Fetch run a zrangebyscore from 0 to current timestamp BUT limit to 5K jobs.
Publish the job and remove it from sorted set

The Issues

This solution actually scaled up until 1-2 years ago when we started having issues with it - the main one's being:

It could not catch up to a huge backlog of delayed messages
It would eventually OOM or SPIKE up to 40GB of memory
1. Due to things being fetched into memory AND some instability OR even internal logic could shovel too much data into Redis - the service just died 💀
We could not scale horizontally - due to consuming and fetching objects into memory before deleting them.

The solution

The solution was very simple: we implemented something that I liked to call streaming approach

Using the same data structure, we are now:

Running a zcount from 0 to current timestamp
- Counting the amount of Jobs -> returning N
Creating an Async Iterator for N times - that used the zpopmin method from Redis
- zpopmin basically returns AND removes the least score object - ie most recent timestamp

The processor for the SortedSet

The Async Iterator

And that's all!

This simple algorithm change annihilated the need for:

Big In Memory fetches - makes our memory allocation big
Limit of 5K in fetches - makes our throughput lower

Results

I think the screenshots can speak for themselves but:

We processed the entire backlog of 40GB of pending jobs pretty quickly
From a constant usage of ~8GB - we dropped down to ~200MB
We are now - trying to be play safe and still oversize - safely allocating 1/4 of the resources.

Money-wise: We are talking at least of 1K USD/month AND more in the future if we can lower our Rediscache instance.

Note

We currently have more enhancements in the roadmap - such as making the job delaying via RPC, using different storages for different postpone amount (1milli, 1 second, 1 day, 1 week++) and making it more reliable overall.

NodeJS - Lazy Init Patterns

Lucas Weis Polesello — Sun, 24 Dec 2023 21:22:51 +0000

One of the main challenges when dealing w/ the async nature of NodeJS is initializing classes/clients that requires some sort of side effect - such as database connection, disk reads or whatsoever. Even the simple idea of waiting for the first use-case to connect/initialize a resource.

Besides Dependency Injection - I like to use two approaches for this:

1) Leaving it up to the client to call connect or any other synonym - easy as creating an async function as the example below

const redis = require('redis');
const crypto = require('crypto');
//PROS: Damn easy, simple and straight-forward

//CONS: This leaves the entire responsibility to the client
class DistributedDataStructure {
    constructor(){
        this.client = redis.createClient();
    }

    async connect(){
        return this.client.connect();
    }

    async add(staffName, reviewId){
        //Do some business here - idk,
        const accountName = await this.client.get(key);
        return this.client.sAdd(`v1:${accountName}:pending-reviews`, reviewId);
    }
}

(async () => {
    const ds = new DistributedDataStructure();
    await ds.connect();
    ds.add('Jerome', crypto.randomBytes(12).toString('hex'));
})()

2) Proxying the access

In the real and wild-world we know that we have to deal w/ legacy code, legacy initialization methods and much more unexpected stuff - for this we have a second use-case which leverages the Proxy API for JS

Using Proxy it would look poorly-like

const redis = require('redis');
const { once } = require('events');
const crypto = require('crypto');

//PROS: No client responsibility - makes it easy for the client
//CONS: More complex and error prone
class ProxiedDistributedDataStructure {
    constructor(){
        this.client = redis.createClient();
        this.client.connect();
        return new Proxy(this, {
            get(target, property){
                const descriptor = target[property];
                if(!descriptor){
                    return;
                }
                if(target.isReady){
                    return descriptor;
                }
                return async function(){
                    await once(target.client, 'ready');
                    return descriptor.apply(target, arguments);
                }
            }
        });
    }

    async add(staffName, reviewId){
        //Do some business here - idk - like below
        const accountName = await this.client.get(staffName);
        return this.client.sAdd(`v1:${accountName}:pending-reviews`, reviewId);
    }
}

const client = new ProxiedDistributedDataStructure();
client.add('Jerome', crypto.randomBytes(12).toString('hex'));

The main benefit for the second approach is that we can instantiate the objects in sync contexts and only treat the method calls as async - instead of needing to play around some dirty gimmicks to call connect and chain promises - even worse, callbackifying.

Dev.to Code Examples

Note: AFAIC from Redis V3^ we have an option legacyMode whenever creating the client which we can keep this lazy nature of Redis - doing client buffering of calls.

Brazilian "Rinha de Backend" challenge #2 - The Improvement

Lucas Weis Polesello — Wed, 30 Aug 2023 12:27:22 +0000

Part #2 of a small series Brazilian "Rinha de Backend" challenge. Click here to part 1

15th of August, 7AM.

I woke up, took a long breakfast, sitting at my office and started tackling the ghost that spent the night before haunting me. (In this case, my brain + overthinking)

Some of his tips, thou, did really help!

Fixing these two queries(1,2) business rules
Adding more replicas into NGINX and DockerCompose
Fine tuning resources on DockerCompose
Fixing PINO async logging
Configuring NGINX worker connections

I was able to get some good improvements reaching ~31% of success which really sounded way better than 10%. But man, something was really off.

Why did my application performed so bad?

Well. I had forgotten to address yesterday's finding. The database bottleneck.

First thing I did was to copy the database query and ran it against at least 40K patients. Surprisingly, it took 50 seconds even thou I was certain I had indexes setup.

But before I explain the database optimization I did I need to explain the structure I had before the improvement.

CREATE TABLE IF NOT EXISTS pessoas (
     id SERIAL PRIMARY KEY,
     apelido VARCHAR(32) UNIQUE NOT NULL,
     nome VARCHAR(100) NOT NULL,
     nascimento DATE NOT NULL,
     stack VARCHAR(32)[]
);

CREATE INDEX IF NOT EXISTS term_search_index_apelido ON pessoas USING gin(to_tsvector('english', apelido));

CREATE INDEX IF NOT EXISTS term_search_index_nome ON pessoas USING gin(to_tsvector('english', nome));

The best tool to analyze what steps the database took to run the query (and much more) is the EXPLAIN.
And of course I had forgotten a index. But how I would index a ARRAY field for FTS? That was a bad design choice. Arrays can't be text indexed, the performance is kinda bad for this use-case.

Well, let's ship this responsibility to the client. The stack field became a JSON field which the client deserializes and serializes. The JSON field is text-indexed and voilá, we now have a index for stack field.

Run the query again: Less than 20ms. Cool. That's what's expected.

Not surprisingly I reached 45% of success and 27% of >1200ms requests. My database lowered CPU usage and errors like premature connection, timeouts and connection losses only appeared by the middle-to-end of the stress test.
Database-wise I thought that It was as far as I could get.

It's time for cache.

I decided to use Redis instead of LRU memory caches since I had distributed applications and I was aiming to have at least 4-5 replicas which would make the same resource not to frequently requested on the same pod/container.

The caching strategy

/pessoas?t
- Cached the response.
POST: /pessoas
- Cached the the entire resource by ID after creation
- Cached the apelido field since it had a unique constraint on the database.
GET: /pessoas/:id
- Checked cache before hitting database.
On the validation middleware I had setup(middleware.js)
- Checked if person already existed on Redis SET and if not checked database. If it did exists, we updated the cache and returned the response back to the client

That is a rather simple Redis caching setup.
(Which I would never ship to production environments at least)

Finally an acceptable success rate! 92% of success being 4% of requests above 1200ms.

But still...That database CPU usage too high even with caching and the connection closes was still a thing.

Back Into DB Optimizations

I figured I had a way over the top PG connection pool configured.

And it was kinda reasonable given that we were running 3-4 replicas, each one with 8 connection on a database that had less than 1 CPU allocated.

(I've also reduce to 3 replicas of the application)

With this small improvement I was able to reach 100% of success rate

Finally I had the same benchmarks from my fellow colleagues but more than so I realized how much the attention to detail needs to be taken seriously when dealing with scale. This was a sandbox experiment and I had a lot of small issues, small legacy-type code (1 day legacy. I'll call it), small premature optimizations and some over the top gimmicks trying to figure out simple stuff.

I had accomplished what I proposed to myself. Although I knew It had way more possible optimization to be done but I decided to step-way from competition.

But this is for the next talk...

Brazilian "Rinha de Backend" challenge #1 - How To Fail

Lucas Weis Polesello — Mon, 28 Aug 2023 13:41:04 +0000

Father's Day at Brazil I received a message from a friend of mine talking about a tech challenge. A quite simple one: Write a API, that has at least 2 replicas, a load balancer and a database, so someone can try to tear it down with a Stress Test.

I though to myself. That is the next level of CRUD developer, so let's try it out.

The initial idea, by which the commit history made it clear I mistakenly called it "finished", was:

NodeJS Express
Postgres
- Indexes for FTS (gin(to_tsvector(lan, field));)
- Stack being a Array field
Batching
- All read requests to database

I knew this configuration was far away from aiming for any TOP 20 but I had confidence it wouldn't fail that much.

After my first NGINX configuration and Docker-Compose YAML created I boldly applied to the challenge.

Sitting at the top of my confidence and not-so worried about it I noticed the competitors publishing screenshots of their own benchmarks.I had realized how much I had underestimated it.

Gatling installed locally, lets give it a shot. To be fair...How bad can my application perform, right?

Lets not talk about how disappointed I got but how worthy it was to try the tool locally.

So, what was happening?

Besides a bunch of premature closed connections, no healthy upstream, 60000ms timeouts, there was no way I my application was bottlenecking that much...

I tried a lot of crazy stuff:

Clustering - via PM2 or native cluster module.
Closing slow requests early
Adjusted some business rules which were off
Reconfigured NGINX
Changing LIBUV process.env.UV_THREADPOOL_SIZE = os.cpus().length

I was very confident I could get at least into 20% of success...Come on...

I had to do something. I was totally blind and didn't knew where to start.

After running a watch "docker stats" I was able to finally get a grasp on it. My database was being hammered.

Time to take it seriously. More than competing with my fellow colleagues I wanted to compete with myself. I knew I could do better. It was a long time I got so excited about something I knew nothing about.

In that late Sunday I already knew Monday would be one of those hyperfocus ADHD days.

Next chapter