I counted 1,726 job postings as 'other'. My classifier was lying.

I counted 1,726 job postings as "other". My classifier was lying, and it moved a headline number.

I publish a free dataset of job postings where the employer states a pay range. To make it more than a CSV, I group every posting into a role family — engineering, sales, product, design, support, and so on — so readers can ask "what does this company pay a data scientist" instead of "what is the average of everything".

Last week the distribution looked wrong in a way that was easy to wave away:

family	postings	share
engineering	3,188	39.7%
other	1,726	21.5%
sales	868	10.8%
support / ops	510	6.3%
data / ml	488	6.1%

One fifth of the corpus landing in "other" is not a labelling preference. It means the classifier did not know, and every per-family median I published was computed on a corpus with a hole in it.

What was actually wrong

The function was one flat list of rules, evaluated in order, with engineering first:

RULES = [
    ("engineering", ("engineer", "developer", "swe", ...)),
    ("data / ml",   ("data", "ml", "machine learning", ...)),
    ("sales",       ("sales", "account executive", ...)),
    ...
]

Two things follow from that shape, and both are silent:

• engineer wins before anything else can be considered. Sales Engineer → engineering. Solutions Engineer → engineering. Support Engineer → engineering. Three different jobs, one bucket, no error message.
• Short, common substrings swallow rare ones. data is inside Data Center Technician, so a rack-and-stack role became a data scientist.

And other was not "jobs we could not classify". It was jobs whose title did not contain any of my keywords — technical program managers, executive assistants, people partners, compliance, fraud. A bucket of 1,726 postings that all mean something different, sitting in the same column.

The fix is boring, and that is the point

Four passes instead of one, ordered by specificity:

1. disambiguation — combinations that a single keyword gets wrong: technical account manager and customer success manager are support, not sales.
2. specific rules — multi-word, unambiguous: account executive, data scientist, platform engineer.
3. generic rules — single strong words: engineer, designer.
4. broad rules — weak signals, tried last: manager, lead.

Ordering is the whole mechanism. engineer only runs after the specific pass has had its chance.

The test caught me, immediately

I wrote 21 assertions before trusting the change, and the third one failed:

AssertionError: 'sales' != 'support / ops'

Technical Account Manager was still landing in sales, because account manager sits in the sales rules and runs first. That is the same class of bug as the original one, one layer down. It is the reason the disambiguation pass exists at all — not because I thought of it, but because a test I wrote to describe my intent disagreed with my code.

What changed in the numbers

Same CSV, only the classifier re-run:

family	before	after
engineering	3,188 (39.7%)	2,200 (27.4%)
other	1,726 (21.5%)	680 (8.5%)
sales	868	1,238
support / ops	510	1,219
finance / legal	268	728
product	319	547

The other bucket did not shrink because I invented categories for it. It shrank because 1,200 postings had been sorted into the wrong family all along — mostly sales engineers, account managers, TPMs and support staff.

The part I did not enjoy: a published claim moved

Three weeks ago I published that, within the same role family, remote roles advertise about 11.5% less pay than on-site ones. That number was computed on top of the broken classifier, with sales engineers counted as engineers.

Re-run on the corrected families, it is −6.2% (remote $212,000 vs on-site $226,000, n=563 vs 1,638).

The direction survived; the size did not. I appended a dated revision note to the original article and wrote the change into the public change log, because a dataset site whose numbers silently drift is worse than one that never published them.

Three checks I now run before publishing any derived column

1. Does the residual bucket have a story? If 20% of rows land in other, that is not a category, it is a bug list.
2. Can two rules both claim the same row? If yes, ordering is load-bearing, and ordering needs a test.
3. Does a change in this column move a headline number? If it can, the change needs a dated note on the article that used it.

The dataset is free and re-generated weekly: salary-data.html, every number that moved is at changes.html, and the code that does the classification is in the same repo.

If you spot a title in the other bucket that clearly belongs somewhere, tell me which family — that is a one-line fix and a test case, and it is exactly the kind of thing that is easier from outside than from inside.