DEV Community: Liu Xiaojun

My open dataset credited the wrong source for 30 numbers — a sort order gave it away

Liu Xiaojun — Sun, 15 Nov 2026 11:42:31 +0000

Last week I published 109 Chinese surnames as open data: JSON and CSV, CC BY 4.0, one record per surname with pinyin, the era it took shape, its founding ancestor, its commanderies, its hall names, its population rank and a prose account of where it came from.

This week I re-released it as v2.1 because the dataset was citing the wrong source for 30 of its numbers.

No value changed. The ranks were already right, the headcounts were already there, and the 109 records are byte-identical apart from one new field. What changed was a sentence in the docs, and the sentence was wrong in a way that made every headcount in the file unreliable to quote. I think that's the more useful story than the dataset itself.

The check that found it

I was drafting a follow-up post about the awkward shape of this data — that only 30 of the 109 surnames have a national population rank, that everything below the official top 100 is somebody's estimate wearing a suit. Before writing it I did the obvious sanity check, the one you'd write in thirty seconds:

import json, urllib.request

rows = json.load(urllib.request.urlopen(
    "https://naming-cn.pages.dev/data/surnames.json"))
ranked = sorted((r for r in rows if r["rank_num"]), key=lambda r: r["rank_num"])

for a, b in zip(ranked, ranked[1:]):
    if b["population_est"] > a["population_est"]:
        print(f'{b["surname"]} #{b["rank_num"]}: {b["population_est"]:,}'
              f'  >  {a["surname"]} #{a["rank_num"]}: {a["population_est"]:,}')

If the ranking is a ranking, a surname ranked lower cannot have more people than the one above it. The loop should print nothing. It printed this:

林 #18: 14,200,000  >  何 #17: 13,000,000
高 #20: 14,500,000  >  罗 #19: 14,000,000
宋 #24: 11,200,000  >  谢 #23: 10,500,000
韩 #27: 9,800,000   >  许 #26: 9,000,000
冯 #29: 9,500,000   >  邓 #28: 9,200,000

Five inversions, all adjacent, all in the same stretch of the table. My first reaction was that I'd fat-fingered five extractions in a row.

It wasn't a typo

I opened each surname page against the JSON. 何 at 13 million and 林 at 14.2 million are both plausible numbers; so are 罗 and 高, 谢 and 宋. Every individual value looked reasonable in isolation. The only thing wrong was their relationship — and only in five places out of 29 adjacent pairs.

That pattern doesn't come from bad typing. Typing errors don't cluster into adjacent pairs and stay plausible. It comes from putting two different columns next to each other and calling them one column.

So I went back to the primary document: the Ministry of Public Security's 2020 national name report, which is the only official Chinese surname statistics at national scale, and which my docs credited for "population ranks and sizes".

What the report actually contains

Here's the part I should have checked before shipping v1.

The report's 百家姓 table ranks the top 100 surnames. It prints the name and the rank. It does not print a headcount. Not "sizes for the top 100" as my docs claimed — the column simply isn't there.

The only figures the report does print are in a separate section on common compound surnames: 欧阳 111.2万, 令狐 5.5万, 诸葛 4.8万, 皇甫 6.4万, 上官 8.8万, and seven more. Twelve of them. And when I compared those twelve against my data, they matched character for character, down to the 万 formatting.

So the file was not a fabrication. It was worse in a subtler way: 12 of the population figures genuinely come from the report, and the other 30 do not. The 30 belong to ranked single-character surnames — 王, 李, 张, 刘, 陈 — whose headcounts come from other surveys and estimates of varying vintage. Some of them are probably close to right. I no longer know which survey each one came from, because the trail goes through sources that all cite the same sentence.

And that sentence is everywhere. Search for any of these numbers and you'll find dozens of pages saying "according to the 2020 Ministry of Public Security report, 李 has 101 million people". They are all repeating an attribution, not a table. A citation repeated often enough starts to look like evidence, and I inherited it the same way everyone else did.

Provenance is not transitive. That's the whole lesson, and I paid for it with a version number.

Why the error was invisible until now

Because the wrong citation is harmless to almost every use of the data.

The ranks are correct. The headcounts are in the right order of magnitude. The only thing that's false is the claim that the two columns share a source — and that claim only matters if you (a) sort by population, or (b) quote a headcount as official. Publish only ranks, and nothing is wrong. Publish only headcounts, and nothing is wrong. Publish both with a single blanket citation, and you've created an error that no reader can see without cross-checking the report themselves.

That's the shape of most data-provenance bugs, I suspect. Not "the number is wrong" but "the number is right and the sentence about it is wrong", which is harder to notice and harder to care about.

The fix: one additive field

v2.1 adds population_source to every record. Four values:

Value	Meaning	Records
`mps2020`	figure printed verbatim in the 2020 report's compound-surname section	12
`estimate`	some other statistic or estimate, vintage varies, not official	30
`qualitative`	only the remark 人数稀少 (recorded as rare), no figure	18
`""`	no data at all	49

Additive only: every v2.0 field keeps its name, order and value, so nobody's parser breaks. If you want only what the report actually says:

import pandas as pd

df = pd.read_csv("https://naming-cn.pages.dev/data/surnames.csv")

official  = df[df["population_source"] == "mps2020"]   # 12 rows, quotable
estimated = df[df["population_source"] == "estimate"]  # 30 rows, cite the caveat

A field is better than a paragraph because a field joins. My old docs had a "sources" section at the bottom that said the ranks and sizes came from the 2020 report. It was prose, so it could be wrong for 30 of 42 records without anything failing. Now the provenance travels in the same row as the number, and the difference between official and not is a filter, not a footnote.

Make the invariant fail loudly

The other half of the fix is unglamorous: the sync script that mirrors the canonical files into the GitHub repo now enforces the semantics of the new field, and prints the inversions on every run.

population_source must be one of mps2020 / estimate / qualitative / ""
mps2020    → must carry a figure, must NOT carry a rank
             (the report's top-100 table has no headcount column)
estimate   → must carry a figure
qualitative→ must carry "人数稀少" and no number
""         → must carry neither

And at the end of a successful sync:

OK 109 records · 30 with a 2020 rank · 42 with a population figure · 18 marked rare
   provenance: 12 mps2020 · 30 estimate · 18 qualitative · 49 none
   note: 5 adjacent inversion(s) when sorting by population_est — 林#18 > 何#17, 高#20 > 罗#19, 宋#24 > 谢#23, 韩#27 > 许#26, 冯#29 > 邓#28
   sort on rank_num; use population_est for magnitude only

The five pairs are printed rather than suppressed, because they are a real property of the data and not a bug to hide. If a sixth ever appears, the numbers have drifted further apart than the dataset is willing to admit, and I want to see it on the next Monday's cron run rather than in a stranger's bug report.

Three things I'd tell anyone publishing a dataset

Ship the provenance in the row, not in a paragraph. A sources section is a promise; a column is a fact. The moment two numbers in the same table have different lineages, they need separate fields, even if that means an "obvious" column that's the same value 109 times.

Write a check that can fail. I read all 109 records while building this dataset and noticed nothing, because reading checks plausibility and plausibility was fine. One loop comparing adjacent rows found the problem in a second. Invariants are cheap; the only cost is admitting you can't tell by looking.

Go to the table, not the sentence about the table. I trusted the sentence, and the sentence was a summary written by someone who also trusted a sentence. The primary document took four minutes to find and settled it immediately.

The data

JSON: https://naming-cn.pages.dev/data/surnames.json (v2.1, 16 fields, 109 records)
CSV: https://naming-cn.pages.dev/data/surnames.csv (UTF-8 with BOM, Excel opens it without mojibake)
Docs: https://naming-cn.pages.dev/en/chinese-surnames-dataset
Mirror + schema + sync script: https://github.com/15905135150/chinese-surnames-dataset

CC BY 4.0. Commercial use, modification and redistribution are all fine; attribution plus a licence link is the only condition.

Corrections are genuinely welcome — and if you know where those 30 headcounts really came from, I would rather have the citation than the number.

I published 109 Chinese surnames as open data — here are the awkward parts

Liu Xiaojun — Fri, 13 Nov 2026 14:49:18 +0000

Update, 2026-11-15. The dataset is now at v2.1 — 16 fields, and a correction to where the population figures come from. It turns out the 2020 MPS report does not print headcounts for the top 100 at all; I had credited it for numbers it never published. No rank or count changed, one field was added, and the story is here: My open dataset credited the wrong source for 30 numbers. The body below is left as it was published, so you can see what v1 claimed.

Every few months someone asks me for a "list of Chinese surnames". The answer is always more annoying than it should be.

The lists that exist fall into three camps. The classic Hundred Family Surnames ordering — 赵钱孙李 — is a Song dynasty poem, not a frequency table, so sorting by it means nothing. Scraped CSVs have thousands of rows, no tone marks, and 欧阳 split into two separate characters somewhere in the middle. Genealogy tables have the detail but usually no licence, or a licence that says "contact us".

So while building a Chinese naming site I extracted the structured parts of my own surname pages and released them: 109 surnames, 13 fields per record, JSON and CSV, CC BY 4.0 (16 fields as of v2.1).

JSON: https://naming-cn.pages.dev/data/surnames.json (87 KB at v2.1)
CSV: https://naming-cn.pages.dev/data/surnames.csv (19 KB, UTF-8 with BOM)
Docs: https://naming-cn.pages.dev/en/chinese-surnames-dataset

Both files are served with Access-Control-Allow-Origin: *, so this works from the browser:

const rows = await fetch('https://naming-cn.pages.dev/data/surnames.json')
  .then(r => r.json());

rows.filter(r => r.rank).length;   // 30
rows.find(r => r.surname === '李'); // full record

No key, no signup. That was the whole point.

What's actually in a record

{
  "surname": "李",
  "slug": "li3",
  "pinyin": "lǐ",
  "era": "商末",
  "ancestor": "李利贞（血缘始祖皋陶）",
  "origin": "河南鹿邑",
  "junwang": ["陇西", "赵郡"],
  "tanghao": [
    { "name": "陇西堂", "note": "以郡望立堂，出自陇西一系" },
    { "name": "青莲堂", "note": "李白自号青莲居士，后世一支取为堂号" }
  ],
  "rank": "全国第 2 位",
  "population": "约 1.01 亿人",
  "notables": ["李耳（老子）", "李广", "李世民", "李白", "李时珍"],
  "origin_note": "嬴姓。舜时皋陶任大理（掌刑狱），子孙以官为氏称「理」。……遂改「理」为「李」。",
  "url": "https://naming-cn.pages.dev/xing/li3"
}

Four decisions in there were less obvious than they look.

1. Compound surnames are one record, not two characters

30 of the 109 are two-character surnames — 欧阳, 司马, 诸葛, 尉迟, 万俟. If you split on characters you get garbage: 欧阳 is not "a surname 欧 followed by 阳", and 万俟 is read mò qí, not wàn sì. So surname holds one or two characters, and slug is the tone-numbered pinyin (ouyang2, moqi2) which doubles as the URL key.

That slug convention is also why lookups don't need tone marks. Want everything starting with li? /^li/ on the slug catches 李 (li3), 梁 (liang2), 廖 (liao4), 林 (lin2), 刘 (liu2), 卢 (lu2), 陆 (lu4), 罗 (luo2), 吕 (lv3) — without you having to type lǚ.

2. Empty means unknown, not zero

Only 30 of the 109 have a population rank. That's not laziness. The Ministry of Public Security's 2020 national name report — the only official Chinese surname statistics at national scale — ranks the top 100 surnames and prints no headcount for them; the figures it does publish cover compound surnames only. Below rank 100 there is no official number at all, and every figure you find online for surname #147 is somebody's estimate presented as fact. (The v2.1 update above unpicks this properly — I had overstated what that report contains.)

So rank and population are empty strings for the other 79, and the docs say plainly: empty means unknown, not zero. If your code does int(r['rank']), it will crash — deliberately. I would rather crash than silently aggregate 79 fake zeros.

3. Arrays are pipe-separated in the CSV

junwang, tanghao and notables are arrays. CSV has no array type, so they're 陇西|赵郡 — a delimiter choice, not a schema. Two consequences:

The CSV flattens tanghao from [{name, note}] to names only, because the note is a sentence and sentences with commas inside a CSV cell are a support ticket waiting to happen. If you need the derivations, use the JSON.
The BOM is there on purpose. Without it, Excel on a Chinese Windows machine opens the file as GBK and every surname is mojibake. The BOM costs you three bytes at the start of the file and one encoding='utf-8-sig' if you're reading it back in Python.

4. Commandery and hall name are not the same field, and merging them loses the point

A junwang (郡望) is a region where a branch of the clan became prominent — the coordinate that determined a family's social rank from the Wei–Jin period through the Tang. A tanghao (堂号) is the inscription on that branch's ancestral hall. Many hall names just repeat the commandery (陇西堂 for the 陇西 branch of 李). Others commemorate an anecdote: 李's 青莲堂 comes from Li Bai calling himself the "Lotus Dweller", and the name carries a claim about poetry and wine rather than about geography.

Flatten both into one "origin" string and you lose exactly the part that's interesting.

The wart I haven't fixed

rank is a display string, "全国第 2 位", not a number. That was fine for rendering the pages and wrong for anyone who wants to sort. The fix is a rank_num field, and it's on the list. For now:

import pandas as pd
df = pd.read_csv('https://naming-cn.pages.dev/data/surnames.csv')
ranked = df[df['rank'].notna()]
ranked['rank_num'] = ranked['rank'].str.extract(r'(\d+)').astype(int)
ranked.sort_values('rank_num').head()

Free-text Chinese in a machine-readable field is the kind of thing you only notice when someone else tries to use your data. Which is an argument for publishing the data in the first place.

Where the numbers come from

Population ranks and sizes: the 2020 MPS report. Commanderies: Tang and Song surname books — the Genealogical Tables of the Chancellors in the New Book of Tang, the Yuanhe Surname Compilation. Hall names: clan genealogies and surviving ancestral halls. The synthesis into one record per surname with a readable origin note is my own work, and errors in it are mine.

Where the sources disagree, the record keeps both readings instead of picking one. ancestor: "白公胜、公子白" is not a typo — it's two documented traditions for the same surname. If you're doing anything analytical with this, filter on origin_note containing 异说 or treat ancestor as potentially multi-valued.

Using it

CC BY 4.0 — commercial use, modification, redistribution all fine; attribution plus a licence link is the only condition.

Chinese Naming (2026). Chinese Surnames Open Dataset.
https://naming-cn.pages.dev/en/chinese-surnames-dataset (CC BY 4.0)

Corrections are genuinely welcome, especially from anyone who knows the genealogy literature better than I do. If a commandery is wrong, I'd rather be told than be cited.