DEV Community

Cover image for Five ways to quietly get a 109-row dataset wrong
Liu Xiaojun
Liu Xiaojun

Posted on

Five ways to quietly get a 109-row dataset wrong

A dataset with 109 rows and 16 fields should not need a user manual. And yet five of those fields will give you quietly wrong answers if you treat them like ordinary columns.

This is a guide to the five traps in the Chinese Surnames Open Dataset, written because I have now watched each of them happen — once to a reader, twice to myself. None of them throw an exception. That's what makes them interesting.

1. slug is a join key, not a pronunciation index

The dataset has both pinyin (lǐ, lǚ) and slug (li3, lv3). The slug is tone-numbered ASCII and doubles as the detail-page URL. It is great for exact joins and terrible for sound-based matching.

A reader asked me why /^li/ only returned five surnames when there are clearly more li-sounding ones. Because the filter is literal:

li3  李      liang2  梁      liao4  廖      lin2  林      liu2  刘
lv3  吕      lu2 卢          lu4 陆          luo2 罗
Enter fullscreen mode Exit fullscreen mode

/^li/ catches the first five. 吕 is lv3 (ü written as v), and 卢/陆/罗 are lu*/luo*. There is no single prefix for a sound.

If you genuinely want sound-based grouping, normalize the tone marks rather than guessing prefixes:

import unicodedata

def sound_key(p):
    # ü and its toned forms become v, then strip combining tone marks
    p = p.translate(str.maketrans({
        "ü": "v", "ǖ": "v", "ǘ": "v", "ǚ": "v", "ǜ": "v",
    }))
    return "".join(
        c for c in unicodedata.normalize("NFKD", p)
        if not unicodedata.combining(c)
    )

sound_key("lǚ")  # 'lv'
sound_key("lú")  # 'lu'
Enter fullscreen mode Exit fullscreen mode

Then group on sound_key(pinyin). But for joins, just use slug exactly. It's the only field guaranteed to be unique and stable across versions.

2. A compound surname is one record, not two characters

30 of the 109 records have a two-character surname: 欧阳, 司马, 诸葛, 尉迟, 万俟. surname is a string, and len() on it is a perfectly reasonable way to find them.

compounds = df[df["surname"].str.len() == 2]   # 30 rows
Enter fullscreen mode Exit fullscreen mode

What breaks is the instinct to take the first character as a key. There is no standalone 欧 in this dataset — only 欧阳 — so a join on 欧 silently drops the row. And 万俟 is read mò qí, not wàn sì, so generating its slug from the characters gives you a key that matches nothing.

The warning extends beyond compounds: several surnames have two readings depending on context. Never build your own key from the display string. slug already exists for that.

3. Null means unknown. Zero means zero people.

This is the one that corrupts aggregates without throwing anything.

Category Records What the population fields hold
mps2020 12 Official figure from the 2020 report's compound-surname section
estimate 30 Other statistics / estimates of varying vintage
qualitative 18 "人数稀少" — rare, no number
empty source 49 Nothing at all

population_est is non-null for only 42 records. The mean of those 42 is about 20.4 million. Fill the other 67 with zero and the mean drops to 7.8 million — a 61% understatement, produced by a single innocent-looking fillna(0).

The right default is to leave the nulls alone and state the denominator:

df["population_est"].mean()                     # 20,365,190 over 42 rows
df["population_est"].fillna(0).mean()            #  7,847,138 over 109 rows — wrong
df["population_est"].count()                     #  42 — say this out loud
Enter fullscreen mode Exit fullscreen mode

The same rule applies to rank_num. Zero is not rank zero; it's no rank.

4. The two population columns are not one column

The newest trap, and the one I found the hard way. rank_num and population_est look like a pair — rank and figure, figure and rank — but they come from different sources. The 2020 MPS report ranks the top 100 surnames and prints no headcounts for them; the headcounts in this file come from other estimates.

Sort by population_est and the order contradicts the official ranking five times:

林 #18: 14,200,000  >  何 #17: 13,000,000
高 #20: 14,500,000  >  罗 #19: 14,000,000
宋 #24: 11,200,000  >  谢 #23: 10,500,000
韩 #27:  9,800,000  >  许 #26:  9,000,000
冯 #29:  9,500,000  >  邓 #28:  9,200,000
Enter fullscreen mode Exit fullscreen mode

So: sort on rank_num. Use population_est for order of magnitude only. And if you need to say "according to the 2020 report", filter first:

official = df[df["population_source"] == "mps2020"]   # 12 rows
Enter fullscreen mode Exit fullscreen mode

Those 12 rows are the only figures in the file that are printed in that report verbatim. (The full story of how that got mislabeled is here.)

5. ancestor and origin_note are multi-valued in disguise

Both look like prose, which means it is tempting to treat them as a single fact. They are not.

Six records carry two documented founding ancestors in one string — 白 is "白公胜、公子白", 吴 is "泰伯、仲雍" — because the sources genuinely disagree. 45 origin_note fields contain 另有 ("another account holds…"), laying out competing branches in parallel. This is deliberate: forcing one root per surname would be a claim the literature doesn't support.

multi_ancestor = df[df["ancestor"].str.contains("、", regex=False)]        # 6
multi_origin   = df[df["origin_note"].str.contains("另有", regex=False)]   # 45
Enter fullscreen mode Exit fullscreen mode

If your pipeline expects one ancestor per surname, split on 、 and keep both values — or keep the string and mark it multi-valued. Do not pick the first one and call it resolved.

Three questions, ten lines

Putting it together:

import pandas as pd

df = pd.read_csv("https://naming-cn.pages.dev/data/surnames.csv")

top10 = (df.dropna(subset=["rank_num"])
           .nsmallest(10, "rank_num")[["surname", "rank_num", "population_source"]])

compound_with_figures = (df[df["surname"].str.len() == 2]
                           .dropna(subset=["population_est"])
                           [["surname", "population"]])       # 12 rows, all official

disputed = df[df["origin_note"].str.contains("另有", regex=False)]  # 45 rows
Enter fullscreen mode Exit fullscreen mode

No regex on Chinese numerals, no guessing at compound boundaries, no invented zeros.

The general version

Every field that can be "unknown" needs a null, not a sentinel. Every number needs its provenance in the same row as the value. Every key needs to be generated, not re-derived from a display string. And every dataset needs one check that can fail — in this case the sync script sorts by rank and refuses to let the five inversions become six.

The machine-readable schema, the validator and a local mirror are in the GitHub repo. Data is CC BY 4.0: use it, modify it, redistribute it, with attribution and a licence link.

Corrections are welcome. If you find a sixth trap, that's a new version number, not a comment.

Top comments (0)