Scrape Wikipedia in 4m♭ (four minutes flat)

I Process All of English Wikipedia in Under 4 Minutes

After a one-time 25 GB download, every subsequent extraction — jazz musicians, sci-fi novels, literary authors, whatever you want — runs against all 25.8 million Wikipedia pages in about three and a half minutes. Here's how I got there.

The goal

Transform a corpus of semi-structured Wikipedia data into searchable, structured database tables as fast as possible.

Background

Previously, I've done data visualization projects with data scraped from Wikipedia pages. I used asynchronous, depth-first page crawling that extracted key data related to jazz music from artist, album, and genre pages.

After fetching a page, a top-level switch classified each one as a jazz artist, jazz album, or jazz-related genre. This determined which shape of Beautiful Soup-based extractor to run on that page, each one suited to target a specific table in the extracted database (artist, album, or genre). To turn a Wikipedia page into a database row, the extractor had to identify the relevant data for each column on that page. In my original architecture, this just meant writing a function to query the DOM for each data element I wanted — such as "name", "instruments", or "years active".

For each data element, I just had to inspect the DOM, write one Beautiful Soup query to grab its value, and cast/map it to the right column. Even accounting for the messiness of real-world wiki page data (DOM structure may not always be the same, or time ranges may look like "1967-2000", "1999-present", or "'62-'89"), this worked fairly well. The vast majority of pages identified at the top level as jazz-related processed without skipping due to parsing errors.

What didn't work

Every time I wanted to change the extract functions or schema to accommodate messier data or new fields, the web crawl could take hours to re-populate the extract database due to network speed and rate limiting.

Even if I could reload already-fetched pages from disk instead of the web, what if I wanted to run extracts on science fiction books instead of jazz music? Caching wouldn't help scale extracts to new subjects.

What could I do to speed things up? Parallelize? Yet, depth-first search is notoriously difficult to parallelize — especially considering the strong coupling of my web-crawling, extraction, and database writing operations.

And yet, most of my system was idling throughout the extract process. That meant there was room to rearchitect and improve. I had proved that I could generate structured data and insights from Wikipedia articles, but how could I design the system to be:

1. Extendable to new topics without re-downloading anything
2. Scalable horizontally, to make full use of my hardware
3. Fast enough that development iterations don't take hours

How I redesigned it

The biggest inspiration came when I noticed my extract jobs were roughly deterministic — I was downloading the same data from the same pages over and over again. This was acceptable for exploratory data analysis, but when it turned into a bottleneck for both the program and my development iterations, I asked: what are the tradeoffs for simply downloading a full dump of wiki data ahead of time?

That question evolved into the architecture below.

The pipeline

graph LR
    F["Dev<br/>writes a job class<br/>(or asks an LLM)"]
    G["Dev<br/>queries the DB<br/>SELECT ..."]
    A["Wikipedia<br/>~25 GB dump"] --> B["Download<br/>(async, one-time)"]
    B --> C["Preprocess:<br/>decompress once<br/>~109 GB raw"]
    C --> D["N workers<br/>parse + match"]
    D --> E["Extracted DB<br/>jazz.db<br/>scifi.db<br/>..."]
    F -.->|"defines tables &<br/>page-to-row mapping"| D
    E -.->|"SQL / pandas"| G

Data from Wikipedia is downloaded from dumps.wikimedia.org, decompressed, and preprocessed as a one-time job. The download takes a couple of hours, but you only repeat it as often as you need fresh wiki data. (Whether or not jazz is dead, I don't need real-time streaming data on the latest album releases.)

Why multiprocessing, not async

The old architecture was I/O-bound — waiting on network requests. The new architecture is CPU-bound — parsing megabytes of wikitext per second. Python's GIL means threading won't help for CPU-bound work, so I use ProcessPoolExecutor to fan out across all available cores.

The key enabler is Wikipedia's multistream dump format. Instead of one monolithic bz2 archive, the dump is split into ~258,000 independently-addressable compressed blocks of ~100 pages each. After a one-time decompression to a raw file, each worker can seek to any block and parse it independently — no coordination, no shared state, no locks.

Jobs are just Python files

Adding a new extraction topic means dropping a single .py file into the jobs/ directory. Each job implements a simple protocol:

class JazzJob:
    name = "jazz"
    default_db_url = "sqlite:///data/databases/jazz.db"

    def matches(self, page: WikiPage) -> bool:
        """Does this Wikipedia page belong to my topic?"""
        ...

    def extract(self, page: WikiPage) -> None:
        """Parse the page and buffer a database row."""
        ...

    def flush(self) -> None:
        """Write buffered rows to the database."""
        ...

    def setup_schema(self) -> None:
        """Create tables from ORM models."""
        ...

No registration, no config files. The CLI auto-discovers every job in the directory:

$ wiki extract list
┌─────────────────┬─────────────────────────────────────────────┐
│ Name            │ Description                                 │
├─────────────────┼─────────────────────────────────────────────┤
│ jazz            │ Jazz artists, albums, and genres             │
│ scifi           │ Sci-fi novels, films, and authors            │
│ literature      │ Literary novels, authors/poets, and poetry   │
│ classical_music │ Classical composers, compositions, and operas│
│ philosophy      │ Philosophers, works, and traditions          │
│ ...             │ (10 jobs total)                              │
└─────────────────┴─────────────────────────────────────────────┘

The results

Here's what actually happens when you run extractions against the full English Wikipedia dump (August 2026, 25.8 million pages) on a Ryzen 9 5950X (16 cores / 32 threads) with 64 GB DDR4 RAM and AIO liquid cooling. With thermal headroom to spare, all 32 threads push to 99–100% utilization for the entire duration of an extract — this is a genuinely CPU-saturated workload:

Extraction timing

Job	Pages Scanned	Matched	Wall Clock	Throughput
jazz	25,792,184	12,441	3m 23s	~196K pages/s
scifi	25,792,184	13,021	3m 22s	~197K pages/s
literature	25,792,184	149,812	5m 04s	~111K pages/s

For lightweight jobs (jazz, scifi), the system sustains roughly 195,000 pages per second across 32 workers. Literature is slower because it matches 12x more pages — 150K vs 12–13K — so the wikitext parsing and database flush overhead dominates.

What you get

Jazz — 10,967 artists, 1,408 albums, 66 genres:

Column	Artists	Albums	Genres
title	100%	100%	100%
name / artist	73.5%	97.0%	—
birth / release year	66.0%	93.0%	—
genres	67.1%	93.8%	—
instruments	53.4%	—	—
labels	35.9%	95.6%	—
personnel	—	80.3%	—
description	—	—	100%

Sci-Fi — 3,233 authors, 6,372 novels, 3,416 films:

Column	Authors	Novels	Films
title	100%	100%	100%
name / author	80.3%	79.3%	—
birth / pub / release year	68.4%	78.0%	82.1%
genres	41.6%	74.5%	—
director	—	—	78.3%
description	—	100%	100%

Literature — 81,514 authors, 66,168 novels, 2,130 poetry collections.

The fill rates reflect the reality of Wikipedia: well-known albums almost always have a label and release year, but many jazz musicians' pages lack a structured infobox for instruments or active years. That's useful signal in itself — it tells you where the data is rich and where you might want to augment it from other sources.

Before and after

graph TB
    subgraph before ["Before: Web Crawling"]
        direction TB
        B1["Fetch page (network)"] --> B2["Parse HTML (Beautiful Soup)"]
        B2 --> B3["Write row to DB"]
        B3 --> B1
        B4["Hours per run"]
        B5["Single topic"]
        B6["Sequential"]
    end

    subgraph after ["After: Dump Processing"]
        direction TB
        A1["Download dump (one-time)"] --> A2["Preprocess (one-time)"]
        A2 --> A3["N workers parse in parallel"]
        A3 --> A4["Per-topic DBs"]
        A5["~3 min per run"]
        A6["Any topic"]
        A7["32 cores"]
    end

	Web Crawling	Dump Processing
Time per run	Hours	~3 minutes
Bottleneck	Network I/O	CPU (wikitext parsing)
New topic	New crawler + hours of fetching	One Python file + 3 min extract
Parallelism	Difficult (coupled I/O + parsing)	Trivial (independent blocks)
Data freshness	Real-time	Monthly/bi-monthly snapshots

The tradeoff is data freshness: dumps are snapshots, not live data. For my use case — historical jazz discographies, literary canons, sci-fi filmographies — monthly snapshots are more than sufficient.

What's next

This architecture is modular enough to support several natural extensions:

• Incremental updates: after a full extract, monitor a targeted subset of pages for changes between dumps rather than re-processing everything
• Richer extraction: use NLP or entity linking to fill in the gaps that simple infobox parsing misses (those red cells in the fill-rate tables)
• More topics: the same dump already supports 10 extraction jobs; adding more is a single file drop-in

Try it yourself

Clone the repo, download a dump, and you'll have a queryable database in minutes:

git clone https://github.com/wright-benjamin-1701/wiki-crawler
cd wiki-crawler
uv sync
uv run wiki dump download --lang en
uv run wiki dump preprocess
uv run wiki extract run jazz       # ~3 min on modern hardware

Then query it however you like — SQL, pandas, whatever. The data is yours.

See the full code on GitHub.