Jeff Lowery

Posted on Jul 22 • Edited on Jul 24

Source Wars: The Chaos of Ten Conflicting Databases

#javascript #database #architecture #chess

Part 2: How I forced ten chaotic, crowdsourced, and uncooperative data sources to speak the same language.

In Part 1, The Transposition Trap, we successfully mapped our chess openings into a Directed Acyclic Graph (DAG) and built 3,600 synthetic bridges to link our orphaned variations. Mathematically, our graph was beautiful.

But content-wise? It was a screaming, chaotic tower of Babel.

To build a truly comprehensive, open-source chess opening database, you can't rely on just one source. You have to aggregate. I pulled data from ten different legendary chess repositories: Lichess, Arasan, ICSBot, ChessTempo, and more.

But when I tried to merge them, I realized that chess players—and the developers who write chess software—have been disagreeing on how to write down names, moves, and codes for decades.

This is the story of how I survived the Source Wars.

The Battleground: Name Fragmentation

You would think that a standardized code system like ECO (Encyclopaedia of Chess Openings) would keep things uniform. It doesn't.

ECO codes (like C89) only cover broad families. The actual names of the variations are left to the whims of whoever is typing them into a database. When merging my ten sources, the same exact position would return completely different strings depending on who wrote the file:

Source A: Ruy Lopez: Marshall Attack, Main Line
Source B: Spanish Game: Marshall Gambit
Source C: Ruy Lopez: Marshall, Modern Variation
Source D: Ruy Lopez, Marshall Blackburne Variation

If I merged these blindly, my UI would cycle through four different names for the exact same board state, giving the user digital whiplash. I needed a single, canonical truth.

To solve this, I wrote a custom sanitizer pipeline that strips out fluff words (like "Game", "System", or "Opening" when they don't add value), standardizes punctuation, and forces a strict hierarchy of [Primary Opening]: [Variation], [Sub-variation].

But that was the easy part. Then, I met the ultimate boss of uncooperative data.

The Ultimate Boss: Wikibooks is Not a Database

Of all ten data sources, Wikibooks was the only one relentlessly hostile to automation. Every other source gave me beautifully structured data: TSV columns, CSV rows, nested JSON trees, or good old PGN.

Wikibooks instead gave me 500 human-edited wiki pages where the URL itself is the data source.

Here is how Wikibooks encodes a chess opening's move history:

[https://en.wikibooks.org/wiki/Chess_Opening_Theory/1._e4/1...e5/2._Nf3/2...Nc6/3._Bb5](https://en.wikibooks.org/wiki/Chess_Opening_Theory/1._e4/1...e5/2._Nf3/2...Nc6/3._Bb5)

White moves use {N}._ {piece}{square}, while Black responses use {N}... {piece}{square}. The URL is the move sequence. The HTML page itself only tells you the opening's name—if you are lucky.

The Regex Pipeline of Doom

Extracting Standard Algebraic Notation (SAN) from these URLs required building a multi-pass regex pipeline just to clean up the formatting:

Strip underscores: Translate 1._e4 into 1. e4.
Drop Black move numbers: Parse 1...e5 and strip the leading numbers to get just e5.
The Castling Trap: Wikibooks editors frequently used zeroes instead of the capital letter 'O' for castling. I had to map 0-0 to O-O on the fly.
URL Artifacts: Scrub out percentage-encoded annotations like %2B (check), %3F (theoretical novelty), and ! (good move).

Wikibooks Editors Are Not Database Engineers

The name you scrape from a wiki heading is whatever a volunteer typed in back in 2008. I found "King's Pawn opening" competing with "King's Pawn Game." I found "Latvian Gambit" fighting "Greco Countergambit."

Worse, many sub-variations completely lacked their parent context. A wiki page heading might say simply "Mengarini Variation", but my database needed the full lineage: "Sicilian Defense: Mengarini Variation".

To keep my sanity, I had to maintain a 4,244-entry alias map (aliases.txt) to manually normalize Wiki names to canonical forms. I also designed a word-overlap algorithm that looks at the parent page, determines if the child heading is missing context, and prepends the parent's opening name—but only if the overlap is insufficient. (For instance, "Latvian Gambit: Mayet Attack" only needs "Mayet Attack" appended, not the full parent string).

Malformed URLs and Dead Ends

When humans write URLs, things break. Eight wiki pages had URLs that completely violated the naming convention. I ran into missing underscores (4.e5 instead of 4._e5), extra periods (4....Nfd7), and a Caro-Kann variation whose URL inexplicably pointed to a London System page. I had to hardcode a corrections.json file to intercept and patch these malformed URLs before the regex engine even looked at them.

I also hit the "anonymous continuation" dead end. Some wiki pages exist solely to sub-divide a long variation, so their heading is just the parent opening's name repeated. The page for:
1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 4. Ba4 Nf6 5. O-O Be7 6. Re1 b5 7. Bb3 d6 8. c3 O-O
...had the heading "Ruy Lopez".

This is identical to the root opening eight moves prior! These are paths the wiki community wanted to document, but never bothered to name. My parser had to detect these by measuring the move-back distance, skipping them entirely rather than polluting the database with duplicate names.

The MediaWiki API as a Change Detector

To avoid hammering Wikipedia's servers, I wrote a change-detection script. The parser queries the MediaWiki API for the last revision timestamp of the Chess_Opening_Theory root page. If nothing has changed, the parser refuses to run.

I took this a step further with a custom tool, diff-wiki.js. It fetches the 500 most recent edits to the category, filters them down to our specific opening pages, deduplicates them, and displays exactly which lines changed. This lets me audit human edits before committing to a full database rebuild.

The Name-Resolution Algorithm

Once the Wikibooks data was scraped, sanitized, and corrected, I merged it back into the master pool with the other nine sources. To resolve conflicts when multiple sources claimed different names for the exact same FEN, I established a strict Source Hierarchy:

const SOURCE_PRIORITY = [
  'lichess',      // Most modern, clean, and player-standardized
  'caissa',       // Solid, historically accurate database
  'wikibooks',    // Rich in variation names, but needs heavy sanitization
  'arasan',       // Great engine book, but terse names
  'openings_txt'  // Legacy fallback
];

When two databases disagreed, the algorithm checked the priority array. If lichess named a position "Sicilian Defense: Old Sicilian", it overrode arasan calling it "KP: Sicilian".

If the conflict was between names within the same priority level, a string similarity threshold (using Jaro-Winkler distance) evaluated the options. If they were 95% similar, it picked the shorter, cleaner name. If they diverged completely, it flagged the FEN for manual review in my corrections.json file.

The Result: One Voice for 17,000 Positions

By building a robust sanitization pipeline, surviving the wild west of Wikibooks URL-scraping, and enforcing a strict source hierarchy, we achieved something incredible.

We took ten fragmented, historical, and crowdsourced files and merged them into a singular, cohesive database. Every single one of our 17,000 graph edges (named openings + interpolated positions now speaks with the exact same voice.

The lesson here is simple: crowdsourced data isn't dirty—it's just data that trusted its authors instead of its consumers. Parsing it means building a solid bridge between human editing habits and machine-readable structures.

Now that our data is clean, unified, and fully connected, we face our final hurdle: How do we fit a 17,000-edge graph into a file small enough to load instantly on a cheap phone?

Watch for Part 3: Size Matters!

DEV Community