DEV Community: IVAO

Why We Moved API Authentication from the Gateway to Our Microservices

David Tchekachev — Tue, 28 Oct 2025 15:05:05 +0000

Moving our API authentication and authorization from gateway to microservice layer

TL;DR

There are various systems design to check if an API call can be processed, we recently moved that logic from our API gateway (aka APIGW) to our microservices.
The main reason being that our APIGW (kong) was facing some performance issues during auth validation, and so creating a bottleneck at the entry point of our infrastructure.

After moving that logic into our microservices, our APIGW is now only focusing on routing the requests (which it does very well), while our APIs are scaling independently based on the load they receive.

A bit of context

Preamble

At IVAO, we built and operate a set of 15 microservice APIs written in NestJS, running on Kubernetes, served by kong as our API Gateway.

Those API services serve over 200 separate endpoints under api.ivao.aero, which require some form of authentication (API Key or JWT Bearer token). Authentication and generic authorization checks were handled by kong, thanks to a Kong custom plugin written in Go, passing headers with needed information to upstream services. The configuration was done using Kong CRD KongPlugin

Here are the most common access rules we have on our API endpoints:

Is the request coming from a user's browser or an autonomous application/bot ?
Is the request coming from an official IVAO website or from a 3rd-party ?
Is the request coming from a staff user ?
Is the request coming from a non-suspended user or application ?

One important information to keep in mind about the control of how our APIs are used: we encourage all IVAO users to build on top of them! Our goal is to expose as much as technically and legally possible for users to build the tools they want to. We even have a website (developers.ivao.aero) where users can generate API Keys and OAuth2 credentials and start querying our APIs within minutes!

The "issues" we were facing

The main issue we were facing was was the plugin’s ability to handle high load. During peak times (typically weekends at IVAO), Kong was processing over 500 HTTP requests per second and often crashed or dropped requests without clear indication of why.. Our main suspect was the custom plugin, based on similar GitHub issues and strange CPU/memory usage reported by its process. We tried scaling Kong from 7 pods up to 15 pods, some of them were still failing...

Our second issue impacted mostly our Web development team as the auth layer wasn't directly in the code they were working on, nor was it in the same language (Go instead of NodeJS), nor could it support very specific rules per endpoint.

Possible solutions ?

Our first and easiest attempt was to optimize options passed to Kong to operate with better performance, as well as review the plugin’s code in depth to fix some memory leaks with database sockets left open. That didn’t help much.

Then we started looking into Kong alternatives, and found multiple options: Envoy, Traefik, etc... But migrating from Kong plugin into those would be a bit painful and wasn't guaranteed to be smooth as we use a single domain and it can point to only one of those Ingress controllers.

Our 3rd option was to use a simple NestJS middleware that would do the same checks the Kong plugin was doing. This solution would make the auth process more maintainable and flexible, while being directly integrated into the NestJS request lifecycle, our internal libs (error messages, logging, caching, database models, etc...). We went with that solution.

Progressive migration

As mentioned in the preamble, we have around 15 separate API codebases, dedicated to various business objectives. Copy-pasting the same code in each of them would be a terrible idea to have a maintainable solution over time.

Luckily, we already had a shared internal library (common-api) that provides utilities, functions, configurations, and plugins reusable across all services.
For example, here is our common NestJS boilerplate code:

  const app = await NestFactory.create<NestFastifyApplication>(
    TopLevelModule,
    new FastifyAdapter({ ignoreTrailingSlash: true }),
    {
      cors: {
        origin: '*', // Allow all origins for CORS
        methods: ['GET', 'POST', 'PUT', 'DELETE', 'PATCH', 'OPTIONS'], // Allow these HTTP methods
        preflightContinue: false, // Do not pass the preflight response to the next handler
        optionsSuccessStatus: 204, // Use 204 for successful OPTIONS requests
      },
      bufferLogs: true,
    },
  );
  const logger = app.get(Logger);

  app.setGlobalPrefix(options.prefix);
  app.useGlobalPipes(new ValidationPipe(TransformationOptions));
  app.useGlobalFilters(new ExceptionsFilter(app.get(HttpAdapterHost)));
  app.useGlobalInterceptors(new AuroraInterceptor());
  app.useLogger(logger);

  // UPDATE: Disabled compression due to RAM usage and CPU load it creates
  // Compression should be registered before static plugin
  // https://dev.to/ivao/why-we-switched-from-nestjs-to-rust-over-one-hidden-and-costly-setting-d9f
  // await app.register<FastifyCompressOptions>(compression, {
  //   // Disable `br` encoding as it takes ages to compress whazzup and other huge response payloads
  //   encodings: ['deflate', 'gzip', 'identity'],
  // });

  await app.register(fastifyStaticPlugin, {
    root: process.env.STATIC_FILES_ROOT_PATH || '/',
  });

  // Backwards compatibility with express for @nestjs/passport https://github.com/nestjs/passport/issues/60
  // Source: https://github.com/nestjs/nest/issues/5702#issuecomment-979893525
  app
    .getHttpAdapter()
    .getInstance()
    .addHook('onRequest', (request, reply, done) => {
      reply.setHeader = function (key, value) {
        return this.raw.setHeader(key, value);
      };
      reply.end = function () {
        this.raw.end();
      };
      request.res = reply;
      request.raw.query = request.query || {};
      done();
    });

  let swaggerOptions = new DocumentBuilder()
    .setTitle(options.title)
    .setDescription(options.description)
    .setVersion(options.version)
    .setContact(
      'IVAO Web Services',
      'https://wiki.ivao.aero/en/home/devops/api/documentation-v2',
      'web@ivao.aero',
    );

  app.useGlobalGuards(new AuthGuard(app.get(Reflector)));

  swaggerOptions = swaggerOptions
    .addBearerAuth(
      {
        type: 'openIdConnect',
        openIdConnectUrl: 'https://api.ivao.aero/.well-known/openid-configuration',
      },
      'oauth2',
    )
    .addSecurity('consumer', {
      type: 'apiKey',
      scheme: 'https',
      in: 'header',
      name: 'apiKey',
    });

  const document = SwaggerModule.createDocument(app, swaggerOptions.build());
  SwaggerModule.setup(`docs/${options.name}`, app, document);

  await enableHealthChecks(app);

  process.on('uncaughtException', (error: Error) => {
    logger.error(error);
  });

  process.on('unhandledRejection', (reason, promise) => {
    promise.catch((err: Error) => {
      logger.error(err);
    });
  });

  process.on('rejectionHandled', (promise) => {
    promise.catch((err: Error) => {
      logger.error(err);
    });
  });

  await app.listen(3000, '0.0.0.0');
  return app;

The implementation was pretty straightforward thanks to NestJS framework.
We added 2 new NestJS middlewares, enabled by default in a non-blocking mode, just adding metadata into the request object passed along:

API Key middleware: Checks for an API Key passed as Header or Query Param, ensure it's valid, and loads associated application's details.
JWT middleware: Checks for a JWT Bearer token present in the Authorization header, verifies the signature as being issued by IVAO's OAuth service, decodes the content before passing it down the NestJS request lifecycle.

Now that we authenticated the request, we need to authorize it by blocking forbidden requests. For this we have added a series of NestJS guards:

User Authentication: Only JWT tokens issued to browser users by our SSO are allowed. Ensures this isn't a bot/application making the request.
Application Authentication: Only API Keys or OAuth2 Client Credentials authentications are allowed. Usually bots and autonomous applications, or internal tools
Official applications: Only IVAO-issued applications are allowed. Especially for sensitive actions and data that we don't want to grant 3rd-party access to
Required OAuth2 Permission: Requires the user to have a specific permission based on their staff position within IVAO.
Required OAuth2 Scope: Ensures the user has granted consent to the application before making the given request. Such as performing actions on their behalf or retrieve personal data.

Thanks to those middlewares being enabled across all our APIs, we can now decorate each endpoint with the guard we want, instead of having a separate config file related to k8s instead of the code itself.

Did it work ?

Code-wise, the migration was pretty smooth and we were able to quickly decommission our custom plugin, and let each API do the auth checks while having the most up-to-date code.

However, we did have to increase the resources on some of the API pods due to this additional logic being added on their pods instead of the central Kong deployment.

Ever since that migration, we didn't experience any issues with kong, which confirmed that moving the auth logic away was the right call.

From TypeScript to SQL: Automatically Granting DB Permissions Without Losing Your Mind

David Tchekachev — Tue, 12 Aug 2025 08:07:36 +0000

TL;DR

Managing database grants across multiple microservice APIs became a repetitive, error-prone task for us at IVAO. Developers often forgot to request new permissions, leading to fragile deployments.

We built a tool extending the Typescript compiler that analyzes the API source code (using AST traversal) to automatically detect which Sequelize models—and therefore which database tables and operations—are actually used. It then generates the exact SQL GRANT statements needed, with zero manual input.

The tool is fully integrated into our CI/CD pipeline and Kubernetes setup, eliminating human error and keeping permissions minimal. We open-sourced it here:

-> ivaoaero/sequelize-grant-generator

A bit of context

Preamble

At IVAO, we partially follow micro-services best practices. This means each business group (e.g., Flight tracking, Flight planning, Air Traffic controlling, Training) has its own NestJS API and ReactJS frontend. But we took a shortcut on the DB side, as we have quite some references across systems, especially for user IDs, airports, etc... We decided to have a single MariaDB instance while having a dedicated logical database for each API, so we still have a separation of concerns but we can guarantee data integrity with foreign keys across those databases.

The "issue" we were facing

Although each API has SELECT, INSERT, UPDATE, DELETE grants on all tables in its own database, we didn’t want to grant these permissions on all databases, as per the principle of least privilege, we only grant what is actually used by the API.

Each time a new feature required new grants, the developer would grant it manually in the staging database to make sure everything worked, and then ask a Tech Lead to do the same in the production database. Now you start understanding how many issues can occur with this workflow...

We needed a deterministic way to manage the DB permissions for each of our APIs to prevent human errors and speed-up our deployment time and migration time when we deploy a new DB instance.

To be honest, this takes us only a few minutes a year, but it's a boring and repetitive task...

Some unsuccessful attempts

Documenting the needed grants

Our first idea was to list all needed grants in the README.md file of the service, so any changes would be easily visible in a Pull Request, hoping the Tech Lead would notice it and make the changes accordingly.

But we anticipated that developers might forget about updating that file and it would quickly end up out-of-sync... Also, it's not a very clean solution. So we decided against it.

Using a 3rd-party tool

On the open-source community, there are many tools that allow you to configure your MariaDB instance with configuration files, as per Infrastructure as Code principle.

With such solution, the developer would need to update a config file (located in the same repository or not) with the needed grants and it would be picked up by the tool that updates the grants automatically.

Although it looked like an elegant solution, it still required developers to manually edit an additional file after working on a feature in the code, and could be forgotten since there was nothing checking its integrity.

A True Source of Truth, Without Extra Work?

Both solutions above forced developers to review the code they worked on, identify the new grants to add or remove, and finally update a separate file to reflect those changes.

I was convinced we could find a way to manage those grants automatically so the developer didn't have to think about it and just import the models he needed in the code.

Over-engineering It (Because Why Not?)

Source

The first question I asked myself: "How can a program know which grants an API needs ?"

The answer was: "Go look in the API's code and figure it out !"

Just as a reminder, our APIs are coded in Typescript with NestJS and Sequelize as our ORM (with the Typescript extension).

My first thought was to write Regexes to extract the models and their usage from the source code, but I knew it would turn into a messy rabbit hole even before I begin.

At this moment I had a flashback from a recent Computer Science course I had: Program Compilers in which we studied the theory behind them and built our own compiler for a simple language (Tiger). There I learnt that compilers actually analyze the whole code and build an Abstract Syntax Tree (AST) that represents the instructions with the correct types, binding, checks, etc...

Indeed, Typescript knows exactly which classes and variables were Sequelize models (as each model extends a shared base Model class) and what methods could be used on them!

One of the key moments from the course that stayed with me, was a quote from the professor:

You can build compiler plugins for any compiled languages to override/extend some actions

-- EPITA professors: Étienne Renaut, Ghiles Ziat, Loïc Blet

Although Typescript isn't a compiled language, it still type-checks the code before transpiling it to JavaScript.

Leverage Typescript type-checking to identify used DB models

In our case, all DB models are defined in a separate NPM package (@ivao.aero/database), which we import in all our APIs, so models can be referenced from other APIs if needed (as explained in the preamble).

In 95% of the case, here is how the code looks like in a given service:

import { Airport, Country } from '@ivaoaero/database';

class CountryService {
    async getCountriesWithAirports() {
        const countries = await Country.findAll({
            /* args */
            include: [Airport]
        })

        // Do some processing

        return countries;
    }
}

For that code, I need my API to have SELECT grants on airports, airport_countries (hidden Many-to-Many table), and countries

By breaking down the problem at hand, I realized I could make it easier by identifying the classes imported at the top of the file, as they would always come from the same package. This is when I found a StackOverflow answer that helped me a lot to lay the foundations for this crazy project.

I also realized that working with AST is very portable from one language to the other, once you understand the theory behind it.

Iterating to improve the tool

At my first attempt, my tool was able to load a Typescript project from the tsconfig.json file, go through each file to get the AST built by the Typescript compiler, and extract all the models imported in the project (ImportDeclaration nodes).
But I still had 2 missing issues:

I didn't know how those models were used. Did the code only require SELECT or needed INSERT, UPDATE, DELETE as well ?
I was missing the hidden Many-to-Many tables that are used when building nested relations, as they aren't directly referenced in the code (whole point of the ORM is to abstract this ;))

I understood that I needed to traverse the whole AST, instead of stopping at the import statements, to find the models usage (CallExpression nodes).

It wasn't as straightforward as processing the imports because you can have some edge-cases like this one:

import { Airport } from '@ivaoaero/database';
class AirportEntity extends PickType(Airport, ['id', 'name']) {}
async func() {
    const instance: AirportEntity = await Airport.findOne(/* args */);
    await instance.update(/* args */);
}

In this case, instance isn't of type Airport but AirportEntity as I have selected only from fields/attributes. My tool wasn't able to understand that .update actually referenced the Sequelize method from the parent class.

I will not go into details on every blocker I encountered but I can confirm it was quite fun to debug Typescript typings to figure out which ones were actual Sequelize models. Also, finding hidden Many-to-Many relations between used models...

After some long evenings, I finally had a working tool !

How does it work ?

Now that the tool is finished, here is how it works: Give it the folder in which the API is stored, and it will return a SQL query which will grant that API's DB user all the grants it needs to function!

For IVAO, we have integrated this tool into our CI/CD pipeline:

When we run yarn build in our pipelines, it also exports the needed grants into a file stored in dist/ which is packaged in the Docker image
In our K8s environment, we added an Init Container that starts just before the API, reads the needed grants from the packaged file, and applies them.

To conclude

With this tool fully integrated into our development process, developers no longer need to think about DB grants—whether adding new ones or cleaning up unused permissions. The codebase itself is now the source of truth for what’s needed at runtime.

We’ve open-sourced it for others who might face similar issues: GitHub sequelize-grant-generator. There, the complete parsing process is documented if you want more details.

Sure, the manual process wasn’t a huge burden. But building this tool? That was the fun part.

We Trust Cargo — But libmysqlclient Taught Us to Check the System Too

David Tchekachev — Tue, 29 Jul 2025 08:00:00 +0000

TL;DR

After bumping some dependencies in our Rust Actix API, we experienced a very strange deserialization issues from our ORM (diesel) on nullable values in an endpoint that used to work perfectly.

Turns out it was due to a version of libmysqlclient installed at OS-level in our containers that wasn't returning the right value in specific cases. It took us a couple of days to figure it out.

A bit of context

Preamble

At IVAO, we need to serve some APIs for a large amount of users: 30M requests / day (avg. 330 rps) with short data liveness: 10 seconds (with plans to reduce it even more).

Of course, we have some caching in place to reduce the load on our APIs and database. For more details, please refer to the post we wrote on our blog

If you are interested in understanding why we have some APIs in NestJS and others in Rust, please refer to our previous article

David Tchekachev for IVAO

Jul 20 '25

Why We Had to Switch from NestJS to Rust Over One Hidden (and Costly) Setting

#nestjs #rust #webdev #api

Comments

5 min read

Some unrelated migration

We were in the process of migrating how our APIs authenticated and authorized our users & applications. Without going into too many details, the JWT validation used to be done by our proxy (Kong) and was being migrated to the end-APIs.

To simplify, the initial plan was:

Add a JWT middleware to our Actix API

Wanting to make it quick & easy, ended up in a rabbit-hole

During the above migration, we needed to import new crates into our project: actix-web-httpauth, jwt-simple-jwks, jwt-simple, nothing too crazy considering what we were doing.

But, here is where it started to get slightly interesting: those new crates have dependencies (in particular base64) that require Rust Edition 2024, while we used to be on Rust Edition 2021, version 1.79.

At this point, our thought was: "Let's bump the edition, rust, and take the opportunity to run cargo update". Shouldn't be too big of a deal, right? At least we thought it would be easier than finding a non-latest version of the crates that didn't require the 2024 edition.

Here are the steps we ended up taking:

We upgraded from Rust 1.79 to Rust 1.87 (latest version available at the time), both in our local environment and in our Docker containers building & running the API.
- As Rust 1.79 didn't support the 2024 edition.
We upgraded from edition: 2021 to edition: 2024
We ran cargo update
We installed the new crates: actix-web-httpauth, jwt-simple-jwks, jwt-simple

It was too easy to be true

At this point, in the local environment, everything worked great and we were able to add the auth middleware we wanted and successfully ran all tests!

We, then, deployed the API to our staging environment to ensuring it integrated well with the services querying it. Although we already tested it locally, nothing beats integration tests in a dedicated environment.

And here the real issue was noticed: We were getting 500's on some of our endpoints!

Starting to dig into the 500's Server Error

When we noticed the 500 response code, we went to our logs, only to find the following error message there:

 DeserializationError(DeserializeFieldError { field_name: Some("arrival_distance"), error: TryFromSlideError(()) })

which comes from our ORM, diesel, that we haven't directly touched during the migration.

Note: arrival_distance is nullable LONG which holds the miles left to destination for a given plane from a reported GPS position. It is returned in the endpoints failing but not used in the middleware at any point.

At first we thought something strange was happening in our staging database and it might be corrupted, explaining the deserialization error. But we didn't find anything there, and running the API locally while being connected to the same database didn't throw any errors.

Looking on Google, nothing was returned for that error message, except the code throwing it ;)

Since the issue was only happening in our staging environment, our first goal was to replicate it locally, which would greatly improve our velocity in solving it, seeing it couldn't be solved quickly.

As much as I tried, I was not able to replicate the issue locally, I checked the rust version, the dependencies, I reinstalled everything, nothing worked...

After some time with unconclusive results, I called for some backup with another developer who had greatly contributed to the project in the past. To my surprise, he was able to replicate the issue on his computer.

At this point, it was the exact representation of It works on my machine, although the changes in the PR didn't contain anything indicating device-specific environment.

We tried reverting some of the changes we did, like uninstalling the new crates (which had nothing to do the deserialization), reverting the edition upgrade, and even downgrading Rust back to 1.79.
But the error was still happening on staging environment but still not reproducible on my computer...

Until then, I was running the API directly from my Linux machine (NixOS) but our staging environment is running the APIs in Docker containers.
Our Dockerfile

FROM rust:1.87-slim-bullseye AS builder
WORKDIR /build

RUN apt-get update -y && apt-get install -y pkg-config libssl-dev default-libmysqlclient-dev

COPY Cargo.toml Cargo.lock ./
COPY crates/ crates/

RUN --mount=type=cache,target=/build/target \
    --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/usr/local/cargo/git \
    --mount=type=cache,target=/usr/local/rustup \
    set -eux; \
    rustup install 1.87.0; \
    cargo fmt --check; \ 
    cargo build --release; \
    objcopy --compress-debug-sections target/release/tracker-api ./tracker-api

FROM docker.io/debian:bullseye-slim

WORKDIR /app

RUN apt-get update -y && apt-get install -y pkg-config libssl-dev default-libmysqlclient-dev

EXPOSE 8000

COPY --from=builder /build/tracker-api ./tracker-api
CMD ./tracker-api

We know the Dockerfile could be improved—but that's a topic for another day 😉_

I decided to try to run the container locally although it wasn't that easy as connecting to services outside the docker network is a bit tricky (connection to staging services to test locally was done with kubectl port-forward which isn't accessible from Docker's network).

Finally, after overcoming the network hurdles, I was able to run the API in a docker containers on my PC, and the error was finally happening on my PC as well! First win!

Trying to find a solution

Now the question was: Why is the API working on my NixOS, but not in Docker ?
Also, How is it related to deserialization ? as we haven't touched that part.

While searching through the deep ends of the internet, I found people trying to bundle the mysqlclient-sys crate directly into the target binary, instead of relying on dynamic linking to the OS-installed version (see Dockerfile above). We tried to do the same thing, but this required quite some additional build tools and still wasn't compiling, so we abandoned that path.

But we kept pursuing that direction, thinking it was indeed related to the MySQL client library. This is when we started looking more closely at the default-libmysqlclient-dev installed in the Docker container, and the one on my NixOS. We realized that we had different versions of that library!

By checking the Debian registry, we confirmed that our Docker container wasn't using the latest version (Debian 11 package version vs Debian 12 package version) while my NixOS was. And we also realized that we hadn't reverted the cargo update we performed during the initial migration, which actually bumped diesel to a new version (2.1.6 -> 2.2.11).

When rebuilding our Docker image with Debian 12 instead of Debian 11 ( :bullseye-slim --> :bookworm-slim), the error was finally gone!

To conclude

From our perspective, it seems like libmysqlclient and/or diesel have made some changes that made some non-latest version incompatible.
Using the latest versions everywhere has fixed the issue.

Although Rust has a package manager (Cargo) that ensures crates are linked to other crates with specific versions, this doesn't concern external libs linked by the OS (libmysqlclient in our case), no version checks are happening to ensure compatibility.

Side Note: Only a few days after discovering this bug, haven't had the time to properly report it, other users have already reported it on GitHub and a fix was submitted quickly afterwards. If we had performed that migration after the bug was discovered and fixed, we would have been able to avoid it or fix it faster.

Side Note²: Looking at the fix applied, it indeed matches our specific case (nullable LONG value). But we wouldn't have been able to pinpoint it as we didn't dig that deep.

Why We Had to Switch from NestJS to Rust Over One Hidden (and Costly) Setting

David Tchekachev — Sun, 20 Jul 2025 23:08:27 +0000

TL;DR

We partially migrated to Rust because NestJS's compression middleware had a default configuration at max level, which noticeably impacted our performance - something that took months to identify and fix.

A bit of context

As presented in our previous post, IVAO has a website displaying live data traffic to thousands of visitors at the same time: Webeye

The website operates fairly simply, it fetches the data every 15 seconds from 3 different endpoints:

/now/pilots: 3 MB
/now/atcs: 3 MB
/now/observers: 1 MB

We also have specific endpoints which are fetched when you click on a specific user:

/sessions/:id: 0.5 MB
/sessions/:id/tracks: 0-5 MB
/whazzup: 5 MB

Each endpoint used to take around 2 seconds to reply, which was unacceptably slow !! We understand NestJS (based on fastify) isn't the go-to framework for low-latency APIs, but it's the easiest to maintain across the 20+ APIs we have, and it's easy to find volunteers with the skills to do so.

Let's try to find the culprit !

First, we checked the NestJS best-practices for performance and confirmed that we were already using fastify instead of express. And we already had compression enabled to reduce the response payload size.

Also, at that time (which isn't the case when I'm writing this article), the authentication & authorization part was handled by Kong, our Kubernetes proxy, so the API would only have to serve the data and not waste compute time on it.

Caching

Most of the time, when an API is slow, the first attempt to speed it up is to cache it.

We have tried to implement caching at CloudFlare level, but it didn't work as all our endpoints are authenticated with an API Key or Bearer token, and Cloudflare doesn't cache those resources (which makes sense).

We have also tried implementing Redis caching which seemed to work well but the first person to make the request would still have to wait those long seconds. While it’s technically possible to pre-populate the cache with a background task, while the API would only be serving it, we decided it would be an ugly and overcomplicated trick that would create new issues.

Digging into the code

While debugging, we leveraged JavaScript’s console.time feature to pinpoint the function taking so much time. Our conclusion was that the request was received and went through all the middlewares pretty quickly, the database query was already pretty fast, but the response was taking most of that time.

There, our first guess that it was related to the serialization of our response which came from Sequelize into JSON.

We have tried to implement Nestia, which is a NestJS variant that should have 200x faster serialization. After some very long nights fighting with it without any improvement, we came to the conclusion that we were either doing something completely wrong, or the issue wasn't in the serialization process...

Out of ideas

At some point, we ran out of ideas, after trying to optimize our NestJS API as much as possible.
As some developers suggested: "Let's try the most state-of-the-art solution for low-latency APIs !"

Migration to Rust (Actix)

Although I wasn't a big fan of introducing a new language into our tech stack (and still am), we decided to give Actix, a Rust framework for APIs, a go; just to see what we could achieve and if the issue was, indeed, coming from NodeJS.

Although we struggled a bit — none of us had experience with the framework, we finally replicated the exact same response bodies as in NestJS. The results were better than what we expected: Endpoints had an average of 150ms ! (From 2s -> 150ms: 12x faster)

Bonus: the Rust API consumed only 50M of RAM, while NestJS consumed over 500M during peak time !! (90% lower memory usage)

There wasn't much to debate, we deployed that Rust API to replace the NestJS one, only for the endpoints frequently accessed by Webeye.

The user experience greatly improved, while our codebase was split across 2 completely different languages. That’s a debate for another day ;)

Being stubborn about keeping NestJS API

Although the Rust API was working very well, it still stayed in mind: "How can NestJS be so slow, and still so widely used", so I kept digging!

During a long night of trying to find a solution, I started noticing something strange:

Requests made from the browser: 2 seconds
Requests made from Postman: 2 seconds
Requests made from curl: 200ms

How is it possible that I wasn't getting the same results from curl ?

Let's try to find the difference between Postman and curl !

I spent a few hours trying to figure out why Postman was so slow at processing requests, only to find that it wasn't Postman's fault....

Postman, by default, sends headers with the request, including Accept-Encoding: Accept-Encoding: br, deflate, gzip, *
Which means that the API will try, in order and if it supports it, to compress the response payload with Brotli, Deflate, Gzip, etc...

When disabling that default header, Postman also started making 200ms requests!
We finally found the culprit!

After a bit of digging on the internet, I found the following GitHub issue on the NestJS repo: perf: brotli compression

Basically, it said that Brotli was configured by default on the max setting, which took time to compress the whole payload

Although the Github Issue recommends setting a lower value to speed up the process, we decided to completely disable Brotli in our APIs as any value we tried to pass would always take longer than the other encodings.

Previous NestJs Bootstrap code:

await app.register<FastifyCompressOptions>(compression);

New NestJs Bootstrap code:

await app.register<FastifyCompressOptions>(compression, {
  // Disable `br` encoding
  encodings: ['deflate', 'gzip', 'identity'],
});

One might ask: "Aren't you trying to save bandwidth to reduce egress fees?", to which we happily reply that we are hosted at OVH in France, and there are 0 egress fees !

With this fix in place, our NestJS APIs are now 10x faster (2s -> 200ms) on big payloads, which is way better than before. With those performances, switching to Rust wouldn't have been worth it (except for the RAM usage - but that's not today's problem).

To conclude

While trying to reduce the response time of our APIs, we enabled NestJS compression middleware, which turned out to be configured at the maximum setting and was the reason our endpoints were taking so long!

How We Handle Megabytes of Real-Time Data in React with IndexedDB

David Tchekachev — Mon, 14 Jul 2025 13:28:43 +0000

TL;DR

We refactored IVAO's live flight map (Webeye) to reduce memory usage and improve performance. By storing & querying fetched data in IndexedDB, and using event-driven rendering, we:

Cut down re-renders by 80%
Improved browser memory footprint

One of our daily struggles

At IVAO, we deliver an aviation simulation network for enthusiasts, our motto being As real as it gets !

If you have an aviation fan in your close circle, or if you are one, you know one thing for sure, they can't live without FlightRadar24 (It's a live map of all planes showing where they are going, the type of airplane, etc...)

Logically, we also have our own version: Webeye, which displays the same information about live flights, online Air Traffic Controllers (ATC), as well as additional data used to better experience the network (e.g., community events, specific sectors).
Unsurprisingly, it’s our most visited website across all the ones we have, we average 50,000 visits per day while rendering approximately 20MB of live data.

Now, the technical challenge is the following: How to display all that information, live, to all the users ?

How it used to be

Webeye is a React frontend, fetching IVAO REST APIs, and leveraging OpenLayers library to render the map with all the overlays (e.g., shapes, planes, clicks, moves, zoom)

The code being a bit old (coded in 2020), it followed the basics of React at the time, without too much complexity, which makes it actually easy to maintain for anyone in the team with some knowledge of React.
In that code, all the data was being fetched from their respective endpoints every 15 seconds (we will cover how we manage that in a separate article), and then stored in global React states because they were referenced in various places (e.g., map rendering, hover popup, details panel, track calculation).
This caused issues like multiple map re-renders when multiple endpoints were refreshed.

That approach actually worked but was very memory intensive (thank you Chrome profiler for pinpointing it), which made the experience laggy...

Delegating data storage and querying to IndexedDB

We started looking into how we could improve the situation and came to the conclusion we needed to reduce the amount of data stored in JavaScript variables.

Major options available in a browser are:

Local Storage / Session Storage: Key-Value string storage
- Works great for simple config values or tokens
- Hard to query
- Data needs to be manually (de)serialized each time
- Storage is limited to 10MB
- Synchronous API
Cookie: Key-Value string storage
- Sent to the server on each request (not needed for us)
- Storage is limited to 4KB
- Synchronous API
IndexedDB: Objects or key-value pairs
- Asynchronous API
- No hard storage limit
- Made for large datasets
- Allows selecting a specific object in a dataset with some filters

Reference Article about browser storages

We decided to implement IndexedDB to see if it would reduce the memory footprint of our web app. To speed up the development, we found a library that wraps up the IndexedDB API into something easy to use: Dexie.js

React + IndexedDB: Event-Driven Rendering

Since IndexedDB is event-based, we decided to follow a similar architecture in our frontend.

On a macro view, here are the major functions we have implemented:

Global data fetching

Every 15 seconds, data is fetched from the API and then directly stored in IndexedDB, from there, the JavaScript variable holding the data is freed as we don't want to pass it to a different context:

async function updatePilotsInIDB(data: PilotSummarySessionResponseDto[]) {
  await dexieDB.pilots.clear(); // Clear data from 15 sec ago
  await dexieDB.pilots.bulkPut(data); // Insert new data
  document.dispatchEvent(new CustomEvent(PILOTS_UPDATE_EVENT));
}

Once the data is persisted in IndexedDB, we dispatch a JavaScript Event which will notify listeners that some new data is available

One might ask why we don't use Dexie's useLiveQuery, it is because this copies the data and stores in a state variable until the next batch comes. Which is exactly what we are trying to avoid, as we need the data only for rendering, then we don't need it in a state anymore, until the next render. Although we still use that hook for some specific data where we filter a specific element we need, and so end up storing a single object of negligible size.

Also, some data doesn't need to be refetched every 15 seconds (e.g., list of airlines), so we took the opportunity to store it without expiration, so the visitor doesn't have to refetch it on each visit.

Map rendering

Now that we have events informing us about underlying data change, we have full control on re-rendering the map with fresh data.

We are trying to batch as much as possible the re-renders, so the client's browser doesn't waste compute resources for nothing.

document.addEventListener(PILOTS_UPDATE_EVENT, batchRender);

Which triggers a re-render only once every 15 seconds, while before we had a re-render per endpoint (x5) every 15 seconds!

Data querying

Some features available on the website require searching for some specific objects across multiple datasets (e.g., search all live sessions for a given user ID, which spans the pilots list, ATC list, observers list).

Having IndexedDB as the underlying storage layer, we can leverage their Index feature, which allows specifying which fields we plan on filtering-on (kinda like NoSQL databases).

export const dexieDB = new Dexie("webeye_db") as Dexie & {
  pilots: EntityTable<PilotSummarySessionResponseDto, "id">;
  atcs: EntityTable<AtcSummarySessionResponseDto, "id">;
  observers: EntityTable<ObserverListSessionResposeDto, "id">;
  airlines: EntityTable<AirlineIDBDto, "icao">;
  notams: EntityTable<NotamIDBDto, "id">;
  specialAreas: EntityTable<SpecialAreaIDBDto, "id">;
  creators: EntityTable<CreatorIDBDto, "userId">;
};
dexieDB.version(1).stores({
  pilots: "id, callsign, userId, flightPlan.arrivalId, flightPlan.departureId",
  atcs: "id, callsign, userId",
  observers: "id, callsign, userId",
  airlines: "icao",
  notams: 'id, *centerIds, *airportIcaos',
  specialAreas: 'id, *centerIds',
  creators: 'userId'
});

Here we created indexes for all the search patterns we needed, which we exploited as shown in this example:

export async function getSessionsByVIDsFromIDB(ids: number[]): Promise<Array<BaseSessionDto>> {
  const [pilots, atcs, obss] = await Promise.all([
    dexieDB.pilots.where("userId").anyOf(ids).toArray(),
    dexieDB.atcs.where("userId").anyOf(ids).toArray(),
    dexieDB.observers.where("userId").anyOf(ids).toArray(),
  ]);
  return [...pilots, ...atcs, ...obss];
}

This way we don't have to load the whole list of online users, instead we have an efficient lookup!

Even on nested fields:

async function getTrafficsAtAirportInIDB(icao: string): Promise<AirportTrafficsDto> {
  const [inbound, outbound] = await Promise.all([
    dexieDB.pilots.where("flightPlan.arrivalId").equals(icao).toArray(),
    dexieDB.pilots.where("flightPlan.departureId").equals(icao).toArray(),
  ]);
  return { inbound, outbound };
}

Visually, this is how it looks like:

So, where are we now ?

Most of the time in programming, it is really hard to say if a change was worth it. In our case, here what we keep from this experience:

Upsides:

The code is a bit cleaner
- We know why something re-renders because we control the events
- Large variables aren't passed across contexts
Better rendering performance
- Since we are batching the updates, they only happen once every 15 seconds instead of 5+ / 15s.
Better memory footprint
- Large states aren't kept after their usage is done (20-30 MB lighter memory profile)
- When we need the complete set, we discard of the variable right after use

Downsides:

A not-so-intuitive approach to React apps.
- Using event-driven updates in React isn't typical at all, some maintainers might get lost

Overall, the experience for our users has improved and allowed us to store persistent data directly in their browser to reduce the requests when reopening the page.

What do you think about our approach ? What would you have done differently ?