Size Matters: Squeezing a 17,000-Edge Graph Into a Mobile-Friendly JSON

Part 3: How I optimized, packed, and compressed a massive chess opening graph to load instantly on a cheap phone.

In Part 1, we tackled the transpositions and bridged the gap between orphaned variations. In Part 2, we fought the "Source Wars" to clean, sanitize, and merge ten uncooperative data sources (including a relentlessly manual Wikibooks parser).

We finally had a masterpiece: a mathematically sound, beautifully normalized, 17,000-edge directed graph representing the entire tree of chess openings.

But then, I ran a build.

And my output directory spit out a single, monstrous, 4.2 megabyte JSON file.

For a backend pipeline, 4.2 MB is a rounding error. But for a mobile-friendly frontend application where users expect instant interactive feedback on every chess move, shipping a multi-megabyte JSON file over a spotty mobile network is a cardinal sin. It means layout thrashing, high latency, and a miserable user experience.

I had to put the graph on a strict diet. Here is how we got @chess-openings/eco.json down to a featherweight fraction of its original size—without writing a single line of complex compression code.

---

Why Chess JSONs are Inherently Bloated

Standard, raw JSON is an incredibly expressive format, but when you represent a massive database of moves and names, it is brutally redundant.

Take a look at two actual entries from our master database. Even though these represent two distinct, specific lines of the French Defense, notice how much text is duplicated:

{
  "rnbqkbnr/2pp1ppp/4p3/1p6/3PP3/8/PP3PPP/RNBQKBNR w KQkq - 0 5": {
    "src": "eco_tsv",
    "eco": "C00",
    "moves": "1. e4 e6 2. d4 a6 3. c4 b5 4. cxb5 axb5",
    "name": "French Defense: St. George Defense, St. George Gambit",
    "aliases": {
      "eco_js": "St. George Defense: St. George Gambit",
      "ct": "St. George Defense, St. Georgs Gambit",
      "chessGraph": "St George's Gambit",
      "icsbot": "St George's Gambit "
    }
  },
  "rnbqkbnr/1ppp1ppp/p3p3/8/2PPP3/8/PP3PPP/RNBQKBNR b KQkq - 0 3": {
    "src": "eco_tsv",
    "eco": "C00",
    "moves": "1. e4 e6 2. d4 a6 3. c4",
    "name": "French Defense: St. George Defense, Three Pawn Attack",
    "aliases": {
      "eco_js": "St. George Defense: New St. George, Three Pawn Attack",
      "ct": "St. George Defense, New St. George",
      "chessGraph": "Three Pawn Attack, St George",
      "icsbot": "Three Pawn Attack, St George "
    }
  }
}

Multiply this by 17,000 positions, and you are wasting megabytes on:

1. Repeating Key Names: Repeating "src", "eco", "moves", "name", "aliases", "eco_js", "ct", "chessGraph", and "icsbot" 17,000 times eats up hundreds of kilobytes of pure ASCII noise.
2. Duplicate Substrings: Because every single chess opening starts from the exact same board setup, the first few ranks of the FEN strings (like rnbqkbnr/) are virtually identical across thousands of keys.
3. Semantic Overlap: Common opening names like "French Defense", "St. George Defense", and "Gambit" appear on dozens of sub-variations.

At first glance, it looks like we need to write a highly complex, custom serialization script to tokenize these strings, map them to integer dictionaries, and "rehydrate" the graph on the client side.

But the best engineering solution is the one you don't have to write.

---

Step 1: Let Gzip Do the Heavy Lifting (via Streams, Not JS Memory)

Instead of over-engineering a complex custom serialization script in Node.js that would eat up system memory loading and processing 17,000 deep objects, we can let standard UNIX utilities and the web's native transport compression do 100% of the work.

I wrote a dead-simple bash script that concatenates the individual JSON segment files (ecoA.json through ecoE.json plus our custom eco_interpolated.json) on the fly.

It treats the files as raw text streams, strips their outer JSON braces using standard sed pipelines, merges them, and pipes the output stream directly into high-level Gzip:

# Strip blank lines, then first and last lines (the outer { and })
strip() {
  sed '/^[[:space:]]*$/d' "$1" | sed '1d;$d'
}

{
  echo '{'
  last_idx=$((${#FILES[@]} - 1 ))
  for i in "${!FILES[@]}"; do
    strip "${FILES[$i]}"
    if [[ $i -lt$last_idx ]]; then
      echo ','
    fi
  done
  echo '}'
} | gzip -9 > "$OUT"

No custom JavaScript transformation, no intermediate combined file, and only modest streaming overhead.

Because our final output has a perfectly flat, highly consistent structure, it is the absolute dream scenario for Gzip's DEFLATE algorithm (which replaces repeating strings with tiny backreference pointers). The repetitive metadata keys ("src", "eco", "moves") and the matching FEN ranks collapse with staggering efficiency.

---

Step 2: Keeping the Client Code Dead Simple

By resisting the urge to build a custom compression format, we kept our frontend and API code unbelievably clean.

Because the database remains a standard, flat JSON key-value map, the frontend doesn't need a "rehydration" step, a parsing library, or any complex state management. It simply loads the JSON, and our lookup logic is a blazing-fast, O(1) direct memory access:

class ChessOpeningDatabase {
  constructor(jsonData) {
    this.db = jsonData;
  }

  // Blazing-fast direct lookup
  getMetadata(fenString) {
    return this.db[fenString] || null;
  }
}

No CPU-heavy iteration, no dictionary mapping, and zero initialization delay when the application boots up on a cheap mobile device.

---

The Payoff: Zero Effort, Massive Savings

By keeping the database flat and letting native server-to-browser compression do its job, we achieved a production-ready payload with absolute minimal friction:

The final payload delivered over the wire to the client's browser is 469 KB—smaller than a single average-sized JPEG image. Yet, it contains a fully functional, mathematically sound, 17,000-edge navigational map of the entire history of chess openings, complete with multiple historical source aliases.

Wrapping Up the Chronicles

This project started with a simple problem: my chess video generator had holes in its openings because our data sources didn't understand transpositions.

By the end of this journey, we:

1. Built a Directed Acyclic Graph (DAG) to map transpositions and engineered synthetic bridges to repair 3,600 "orphan" positions.
2. Created a normalization sanitizer to force ten chaotic, historical databases to speak with one voice, culminating in a custom scraper to tame a relentlessly uncooperative Wikibooks.
3. Delivered the entire system as a 469 KB production payload by leaning into structural repetition and native browser compression.

Software engineering isn't just about making things work. Sometimes, it is about knowing when not to write code. By letting standard web protocols do what they do best, we kept our codebase simple, our payload featherweight, and our user experience flawless.

See It in Action

The compressed graph is not merely a benchmark artifact. It powers Fenster, an opening-research web app that lets you play through moves, follow transpositions, and explore the opening names attached to each resulting position.

The opening graph at work in Fenster: one position, several continuations, and a transposition into another opening family. Click the image for a larger view.

[Explore this position in Fenster]

Fenster is the practical test of the architecture described in this series: the graph must be complete enough to navigate, consistent enough to explain what the user is seeing, and small enough to load without getting in the way.

What began as a collection of conflicting opening records is now a usable research tool—one that can trace how openings branch, converge, transpose, and acquire their names.