TL;DR
We refactored IVAO's live flight map (Webeye) to reduce memory usage and improve performance. By storing & querying fetched data in IndexedDB, and using event-driven rendering, we:
- Cut down re-renders by 80%
- Improved browser memory footprint
One of our daily struggles
At IVAO, we deliver an aviation simulation network for enthusiasts, our motto being As real as it gets !
If you have an aviation fan in your close circle, or if you are one, you know one thing for sure, they can't live without FlightRadar24 (It's a live map of all planes showing where they are going, the type of airplane, etc...)
Logically, we also have our own version: Webeye, which displays the same information about live flights, online Air Traffic Controllers (ATC), as well as additional data used to better experience the network (e.g., community events, specific sectors).
Unsurprisingly, it’s our most visited website across all the ones we have, we average 50,000 visits per day while rendering approximately 20MB of live data.
Now, the technical challenge is the following: How to display all that information, live, to all the users ?
How it used to be
Webeye is a React frontend, fetching IVAO REST APIs, and leveraging OpenLayers library to render the map with all the overlays (e.g., shapes, planes, clicks, moves, zoom)
The code being a bit old (coded in 2020), it followed the basics of React at the time, without too much complexity, which makes it actually easy to maintain for anyone in the team with some knowledge of React.
In that code, all the data was being fetched from their respective endpoints every 15 seconds (we will cover how we manage that in a separate article), and then stored in global React states because they were referenced in various places (e.g., map rendering, hover popup, details panel, track calculation).
This caused issues like multiple map re-renders when multiple endpoints were refreshed.
That approach actually worked but was very memory intensive (thank you Chrome profiler for pinpointing it), which made the experience laggy...
Delegating data storage and querying to IndexedDB
We started looking into how we could improve the situation and came to the conclusion we needed to reduce the amount of data stored in JavaScript variables.
Major options available in a browser are:
-
Local Storage / Session Storage: Key-Value string storage
- Works great for simple config values or tokens
- Hard to query
- Data needs to be manually (de)serialized each time
- Storage is limited to 10MB
- Synchronous API
-
Cookie: Key-Value string storage
- Sent to the server on each request (not needed for us)
- Storage is limited to 4KB
- Synchronous API
-
IndexedDB: Objects or key-value pairs
- Asynchronous API
- No hard storage limit
- Made for large datasets
- Allows selecting a specific object in a dataset with some filters
Reference Article about browser storages
We decided to implement IndexedDB to see if it would reduce the memory footprint of our web app. To speed up the development, we found a library that wraps up the IndexedDB API into something easy to use: Dexie.js
React + IndexedDB: Event-Driven Rendering
Since IndexedDB is event-based, we decided to follow a similar architecture in our frontend.
On a macro view, here are the major functions we have implemented:
Global data fetching
Every 15 seconds, data is fetched from the API and then directly stored in IndexedDB, from there, the JavaScript variable holding the data is freed as we don't want to pass it to a different context:
async function updatePilotsInIDB(data: PilotSummarySessionResponseDto[]) {
await dexieDB.pilots.clear(); // Clear data from 15 sec ago
await dexieDB.pilots.bulkPut(data); // Insert new data
document.dispatchEvent(new CustomEvent(PILOTS_UPDATE_EVENT));
}
Once the data is persisted in IndexedDB, we dispatch a JavaScript Event which will notify listeners that some new data is available
One might ask why we don't use Dexie's useLiveQuery, it is because this copies the data and stores in a state variable until the next batch comes. Which is exactly what we are trying to avoid, as we need the data only for rendering, then we don't need it in a state anymore, until the next render. Although we still use that hook for some specific data where we filter a specific element we need, and so end up storing a single object of negligible size.
Also, some data doesn't need to be refetched every 15 seconds (e.g., list of airlines), so we took the opportunity to store it without expiration, so the visitor doesn't have to refetch it on each visit.
Map rendering
Now that we have events informing us about underlying data change, we have full control on re-rendering the map with fresh data.
We are trying to batch as much as possible the re-renders, so the client's browser doesn't waste compute resources for nothing.
document.addEventListener(PILOTS_UPDATE_EVENT, batchRender);
Which triggers a re-render only once every 15 seconds, while before we had a re-render per endpoint (x5) every 15 seconds!
Data querying
Some features available on the website require searching for some specific objects across multiple datasets (e.g., search all live sessions for a given user ID, which spans the pilots list, ATC list, observers list).
Having IndexedDB as the underlying storage layer, we can leverage their Index feature, which allows specifying which fields we plan on filtering-on (kinda like NoSQL databases).
export const dexieDB = new Dexie("webeye_db") as Dexie & {
pilots: EntityTable<PilotSummarySessionResponseDto, "id">;
atcs: EntityTable<AtcSummarySessionResponseDto, "id">;
observers: EntityTable<ObserverListSessionResposeDto, "id">;
airlines: EntityTable<AirlineIDBDto, "icao">;
notams: EntityTable<NotamIDBDto, "id">;
specialAreas: EntityTable<SpecialAreaIDBDto, "id">;
creators: EntityTable<CreatorIDBDto, "userId">;
};
dexieDB.version(1).stores({
pilots: "id, callsign, userId, flightPlan.arrivalId, flightPlan.departureId",
atcs: "id, callsign, userId",
observers: "id, callsign, userId",
airlines: "icao",
notams: 'id, *centerIds, *airportIcaos',
specialAreas: 'id, *centerIds',
creators: 'userId'
});
Here we created indexes for all the search patterns we needed, which we exploited as shown in this example:
export async function getSessionsByVIDsFromIDB(ids: number[]): Promise<Array<BaseSessionDto>> {
const [pilots, atcs, obss] = await Promise.all([
dexieDB.pilots.where("userId").anyOf(ids).toArray(),
dexieDB.atcs.where("userId").anyOf(ids).toArray(),
dexieDB.observers.where("userId").anyOf(ids).toArray(),
]);
return [...pilots, ...atcs, ...obss];
}
This way we don't have to load the whole list of online users, instead we have an efficient lookup!
Even on nested fields:
async function getTrafficsAtAirportInIDB(icao: string): Promise<AirportTrafficsDto> {
const [inbound, outbound] = await Promise.all([
dexieDB.pilots.where("flightPlan.arrivalId").equals(icao).toArray(),
dexieDB.pilots.where("flightPlan.departureId").equals(icao).toArray(),
]);
return { inbound, outbound };
}
Visually, this is how it looks like:
So, where are we now ?
Most of the time in programming, it is really hard to say if a change was worth it. In our case, here what we keep from this experience:
Upsides:
- The code is a bit cleaner
- We know why something re-renders because we control the events
- Large variables aren't passed across contexts
- Better rendering performance
- Since we are batching the updates, they only happen once every 15 seconds instead of 5+ / 15s.
- Better memory footprint
- Large states aren't kept after their usage is done (20-30 MB lighter memory profile)
- When we need the complete set, we discard of the variable right after use
Downsides:
- A not-so-intuitive approach to React apps.
- Using event-driven updates in React isn't typical at all, some maintainers might get lost
Overall, the experience for our users has improved and allowed us to store persistent data directly in their browser to reduce the requests when reopening the page.
What do you think about our approach ? What would you have done differently ?
Top comments (0)