I wanted a clean table of Egypt's pharaohs: who ruled, in which dynasty, for how long, and which pyramid belongs to whom. Wikidata has most of it, but "most of it, as linked data" is a long way from a table you can trust. So I built it the way I would build any production dataset: extract, clean, validate, and keep a record of every value the pipeline refused.
The result is three tables (513 pharaohs, 38 dynasties, 84 pyramids), published on Kaggle and Hugging Face, with the code on GitHub.
The pipeline
extract.py three SPARQL queries against query.wikidata.org, one a minute
-> raw JSON + a manifest with the query text and the retrieval time
build.py raw rows -> tables (dates, precision flags, one row per entity, chronology join),
out-of-range dates nulled and logged
validate.py pandera schemas for every table; the build fails if a check fails
Two habits from production work made the biggest difference.
Keep the raw pull and its provenance. Every run saves the raw query results next to a manifest with the exact SPARQL text and the time it ran. When a number looks odd, I can trace it back to the query that produced it, and when Wikidata improves, I re-run the extract and diff.
Validate as part of the build, not afterwards. The pandera schemas check that ids are unique, years fall between 3300 BC and AD 400, reigns end after they start, pyramid coordinates fall inside Egypt, heights are positive and URLs look like URLs. A failed check stops the build.
Dates are the hard part
Wikidata stores years in astronomical numbering, where year 0 exists. Its -1549 is 1550 BC, not 1549 BC. The tables keep both: a signed astronomical year that sorts and subtracts correctly, and a label that reads "1550 BC".
Precision matters just as much. Wikidata records whether a date is known to the year, the decade or the century, and a reign known to the century is not a reign known to the year. Every date column carries its precision, so nobody computes a reign length from two century-level guesses without noticing.
Some values had to go. The validation caught legendary reigns dated to "4001 BC" and similar impossible dates. Rather than silently dropping them, the build nulls the value and writes a row to quality_notes.csv saying what was removed and why. There are 15 such notes. A quality log like this is cheap to keep and it is the first thing I would want as a user of someone else's dataset.
Filling the gap Wikidata leaves
Wikidata is thin on dynasty dates, so I added a hand-curated chronology: the conventional dates from Shaw's Oxford History of Ancient Egypt (2000), joined to Wikidata's own records. It is one scheme among several, and dates before the New Kingdom carry decades of uncertainty; the file says so in a note column rather than pretending otherwise.
What the data can and cannot tell you
- Wikidata assigns a dynasty to 384 of the 513 pharaohs and has reign dates for only 76.
- The Second Intermediate Period alone contributes about 150 names, mostly kings known from a single scarab or one entry in a king list.
- 302 pharaohs have an image on Wikimedia Commons, which makes the table usable for a gallery or a quiz.
- Pyramid heights are sparse (17 of 84). The literature has them; Wikidata does not yet.
Reuse it
The extract takes about three minutes because of the polite one-query-a-minute rate limit; the build and validation take seconds:
pip install -r requirements.txt
python pipeline/extract.py
python pipeline/build.py
The code is MIT-licensed. Data from Wikidata is CC0, and the curated chronology and derived tables are CC BY 4.0. Images are linked, not copied; each keeps its own licence on Wikimedia Commons.
- Code: github.com/engomaressam/pharaohs-of-egypt
- Kaggle dataset: engomaressam/pharaohs-of-egypt
- Hugging Face dataset: omaressamrme/pharaohs-of-egypt
Omar Essam El Sherbiny. Head of Data at WalaPlus. Data platforms on Google Cloud, data governance, applied ML. More at omaressam.web.app.
Top comments (0)