text === '' is the wrong way to check whether Unicode normalization ate your data

qbofile converts bank files between the formats accounting
software uses. One of its output formats, QBO, declares its encoding in the file header:

ENCODING:USASCII
CHARSET:1252

So every payee name has to be reduced to printable ASCII before it goes in. The function
that does it runs NFKD, strips the combining marks, applies a fallback map for the
characters NFKD cannot decompose, and replaces whatever is still outside printable ASCII
with a space:

sanitizeText('CAFÉ')        // 'CAFE'
sanitizeText('MCDONALD’S')  // "MCDONALD'S"
sanitizeText('北京 STORE')   // 'STORE'
sanitizeText('北京烤鸭')      // ''

Two of those outcomes are fine. CAFE is still readable, and so is the straightened
apostrophe. The last one is not: there is no ASCII spelling of 北京, so the characters
become spaces, the spaces collapse, trim() finishes the job, and the field is empty.

Those two outcomes need different handling. Transliteration should be silent, because a
warning that fires on every é is a warning people learn to dismiss. Losing the whole
name should not be silent, because the user ends up with a transaction that imports
cleanly, with the right date and the right amount, and nothing in the payee field.

So the warning counted the two cases separately:

const { text, dropped } = sanitizeCore(src);
if (!dropped.length) continue;
if (text === '') emptied += 1;
else partial += 1;

That reads like it is obviously correct. It is not, and the input that breaks it is
ordinary.

The input

sanitizeText('北京，上海')   // ','

Not an empty string. One comma.

I did not put that comma there. The input contains U+FF0C FULLWIDTH COMMA, the comma used
in Chinese and Japanese text. NFKD is a compatibility decomposition, and fullwidth
forms are exactly the kind of thing compatibility covers:

'，'.normalize('NFKD')   // ','   U+FF0C becomes U+002C
'（'.normalize('NFKD')   // '('
'）'.normalize('NFKD')   // ')'
'：'.normalize('NFKD')   // ':'
'！'.normalize('NFKD')   // '!'
'　'.normalize('NFKD')   // ' '   ideographic space becomes an ordinary space

The name is gone. The punctuation between the parts of the name is not, because the
punctuation has an ASCII twin and the characters do not.

So text === '' is false, the else branch runs, and the user is told this:

1 description lost characters. QBO files can only hold plain ASCII text. Accented
letters were converted (café becomes cafe), but these characters have no ASCII
equivalent and had to be dropped: 北 京 上 海. Amounts and dates are unchanged. Use
the CSV or PDF output if those characters matter.

Every sentence there is true. It is still the wrong message, because it is the message for
a name that lost some of its characters, and the line that belongs to a name that lost
all of them never gets printed:

A transaction like that still imports with the right date and amount, but with no payee
name on it.

The blind spot was shaped exactly like a Chinese or Japanese business name with
punctuation in it:

北京，上海            ->  ','
星巴克（北京）         ->  '( )'
全家便利店：朝阳店      ->  ':'
株式会社（東京）       ->  '( )'

Four names. Four survivors, none of them a letter.

The case that gets reported is the other one, where the whole string reduces to ''. It
is a known shape: slug builders that return an empty slug for a CJK title, sanitizers that
hand an empty name to an API which requires a non-empty one. Issues get filed about it,
and the guard people write for it is exactly the guard I had written.

This is the sibling case, and it is the worse of the two. The string is not empty, so the
guard passes. Nothing downstream has any reason to look again.

The fix

The question was never whether the string is empty. It was whether anything is left that a
person could read as a name:

if (!/[A-Za-z0-9]/.test(text)) emptied += 1;
else partial += 1;

Punctuation is not content. Digits are, because 全家 #123 reduces to #123, which is a
bad name but still something you can match against a line on your statement.

Checking that the new line holds

Two mutations, against a suite of 495 tests:

if (text === '')              the old check, put back      494 pass, 1 fail
if (!/[A-Za-z]/.test(text))   digits dropped from the      494 pass, 1 fail
                              definition of content

The first mutation is the bug itself. The test that catches it feeds in 北京，上海,
星巴克（北京） and 全家便利店：朝阳店, and asserts that the message says the payee name
is gone and does not say lost characters.

The second one matters more to me. It is the kind of edit that looks like a tidy-up:
names are made of letters, so why is a digit in there. It fails on 全家 #123,
7-ELEVEN 北京 and 北京（STORE）, which all keep something legible and must not be
reported as emptied.

A suite that only catches the exact bug you already fixed is a regression test. One that
also catches a plausible future edit to the same line is closer to a specification.

The part that travels

A normalization step can turn characters you cannot represent into characters you can
without carrying any of the meaning across. Once that has happened, the length of the
output tells you nothing about how much survived.

wrong    did normalization leave anything behind
right    did normalization leave anything that still carries the meaning

It is worth looking anywhere you normalize and then test for emptiness: slugs, search
keys, dedup keys, cache keys, generated filenames. A CJK article title that slugifies to
- is not an empty slug, and every === '' guard downstream will happily agree that
nothing went wrong.

One note for anyone who arrived at this from the accounting side rather than the Unicode
side: this entire class of problem exists because the target format declares US-ASCII in
its header. QuickBooks Online reads a bank CSV directly
when the columns line up, and nothing on that path sanitizes anything.