Inside a folder on my desktop there is a directory called yün (Turkish for wool). From that folder, typing cd yün walks right into it. From the same place, find . -name "*yün*" returns nothing at all. The same three letters, the same keyboard, both times. And there are eight visible files and directories there with yün in the name.
This is the wall every person with non-ASCII filenames eventually hits, and the culprit is not where you would look for it. The file system is working perfectly. The flaw is that every layer above it treats a filename as text. To the file system, yün is an identity. To grep, it is four bytes as it comes off the keyboard and five as it sits on disk. The whole story lives in that one byte.
This article traces where the gap comes from, which tool stands on which side of it, and how to get it right in your own code. Along the way, two steps of a recipe in Apple's own documentation turn out not to hold.
The lab
Every output below comes from this machine, on 19 November 2026:
macOS 26.6.2 (build 25G83), Apple silicon
diskutil apfs list -> "Mustafa (Case-insensitive)" [APFS]
zsh 5.9 (arm64-apple-darwin25.0)
/bin/bash 3.2.57(1)-release (no newer bash on this system)
git 2.48.1
/usr/bin/rsync -> openrsync: protocol version 29 / "rsync version 2.6.9 compatible"
locale -> LANG empty, LC_CTYPE="C"
That last line matters. I repeated the shell comparisons below under LC_ALL=C, en_US.UTF-8 and tr_TR.UTF-8 separately: three locales × two shells × two spellings, twelve runs, all in agreement. So nothing here is a locale artifact. The reason for pinning the environment first: a PATH comparison on this blog once produced 43 phantom differences purely from an inherited environment.
Which Turkish letter is at risk, and which is not
Unicode lets some letters be written two ways: as a single code point (NFC, "composed") or as a base letter plus a combining mark (NFD, "decomposed"). Turkish's seven distinctive letters are not equally exposed:
ç U+00E7 -> NFD: U+0063 U+0327 (c + cedilla)
ğ U+011F -> NFD: U+0067 U+0306 (g + breve)
ş U+015F -> NFD: U+0073 U+0327 (s + cedilla)
ü U+00FC -> NFD: U+0075 U+0308 (u + diaeresis)
ö U+00F6 -> NFD: U+006F U+0308 (o + diaeresis)
İ U+0130 -> NFD: U+0049 U+0307 (I + dot above)
ı U+0131 -> NFD: U+0131 (single code point, DOES NOT DECOMPOSE)
So the word ışık is completely immune to this problem and ışığın is not. Dotless ı has no mark to shed, because Unicode defines it as a letter in its own right rather than a letter wearing an accent. Turkish's most argued-about character turns out to be the only carefree one here. The word çalışma is 7 code points in NFC and 9 in NFD; the difference is two, because ç and ş decompose while ı holds its ground.
The only thing to carry forward: two names that look identical can differ byte for byte.
What APFS does: keeps the name, loosens the lookup
Start with the file system's side, because nothing is broken there. In an empty directory I created a çalışma.txt with the NFC spelling, then tried to open it with the NFD spelling:
>>> open(nfc, "w").write("NFC ile yazildi\n")
>>> open(nfd).read()
'NFC ile yazildi' # found it
>>> open(nfd, "w").write("NFD ile yazildi\n")
>>> len(os.listdir("."))
1 # no second file appeared
>>> open(nfc).read()
'NFD ile yazildi' # it overwrote the same file
One file in the directory, and the name on disk is still NFC — the form supplied at creation. The reverse holds too: creating ağ.txt with the NFD spelling in an empty directory leaves NFD on disk — 7 code points, 8 bytes, where the NFC spelling would have been 6 code points and 7 bytes. APFS does not fix up the form, it stores what it is given; it merely treats both forms as the same name during lookup.
Apple's own documentation says so plainly. The archived APFS guide states that APFS "preserves both case and normalization of the filename on disk in all variants", that lookups "use hashes of the normalized form of the filename", and that a filename "can be found with any of its normalization variants". HFS+ worked differently: it "stores the normalized form of the filename on disk" — the old file system rewrote the name, the new one leaves it alone.
The final sentence of that same paragraph is what this article is really answering: "This means that you don't need to do any additional work to ensure correct normalization behavior." Apple is saying you have nothing extra to do. That sentence is true at the file system boundary — and it collapses one step above it. The rest of this article is the measurement of exactly that.
The page is also stamped "Retired Document", last updated 4 June 2018. The more current and more technical source is Apple's file system reference (22 June 2020), where the behaviour appears as a volume flag:
#define APFS_INCOMPAT_NORMALIZATION_INSENSITIVE 0x00000008LL
/* "Filenames on this volume are normalization insensitive." */
#define APFS_INCOMPAT_CASE_INSENSITIVE 0x00000001LL
Two steps of the documented hash recipe do not hold
That same reference also gives the six steps that produce a directory record's 22-bit hash: take the name as UTF-8, normalize it with canonical decomposition (NFD), represent it as a "null-terminated UTF-32 string", compute CRC-32C, complement the bits, keep the low 22. The document further states that readdir(2) on APFS returns names in hash order. That is a testable claim: apply the recipe, sort by it, and the real directory listing should come back.
I created ten files, a.txt through j.txt. The real listing is not in dictionary order:
readdir: j.txt i.txt h.txt c.txt b.txt a.txt e.txt d.txt f.txt g.txt
Applying the recipe literally, the order did not match. Changing two things made it match:
def apfs_hash(name):
s = unicodedata.normalize("NFD", name.lower()) # (2) lowercasing is NOT in the doc
b = b"".join(ord(c).to_bytes(4, "little") for c in s) # (1) NO null terminator
return (~crc32c(b)) & 0x3FFFFF
predicted: j.txt i.txt h.txt c.txt b.txt a.txt e.txt d.txt f.txt g.txt
observed : j.txt i.txt h.txt c.txt b.txt a.txt e.txt d.txt f.txt g.txt -> identical
To rule out coincidence, two more sets: nine mixed-case names (Rapor.txt, gAMMA, ZIP.bin…) and eight Turkish names (çalışma.txt, GÜNEŞ.dat, İSTANBUL.txt, ıhlamur…). Both reproduced exactly — but only with the lowercasing step. Drop it and the mixed-case set breaks.
So there are two deviations, neither written down:
- The NUL terminator is not included. The document says "null-terminated UTF-32 string"; every attempt that appended the four-byte terminator failed to match, and removing it made the order fall into place.
-
The name is lowercased first. On a case-insensitive volume that is unsurprising, yet it appears nowhere in the hash recipe. One detail for Turkish readers:
İSTANBUL.txtmatched the output of Python's locale-independentlower(), so the folding follows Unicode's general rule rather than Turkish's i/ı rule.
Two things follow. A directory listing really is in hash order — so do not let your code assume it is sorted. And you would never see the gap in a six-year-old document unless you tried to reproduce it.
The layer above: opening works, searching doesn't
Here is the part that bites. A single çalışma.txt sits on disk stored as NFD. In that directory I tried every route using the NFC spelling of the same word:
cat çalışma.txt -> SUCCESS (read the contents)
[ -e çalışma.txt ] -> exists
find . -name "*çalışma*" -> 0 results
ls | grep çalışma -> 0 results
python3 glob.glob("*çalışma*") -> 0 (with the NFD pattern: 1)
node readdirSync + includes -> 0 (with the NFD pattern: 1)
The line runs straight down the middle. Everything that asks the file system finds the file: open, stat, test -e, cat. Everything that compares the name itself misses it: find -name, grep, glob, your own includes call.
Reading that split as "this tool is right, that one is wrong" would mislead you, because the tools are not the issue — the question is. A filename lives two lives: an identity inside the file system, and a string the moment it steps outside. open asks about the identity, find asks about the string. They usually return the same answer, which is why we take them for one question; in a Turkish filename they come apart. The rule is better stated this way: only the file system can answer an identity question, and no layer above it can.
My shell re-encodes the name as it reads it
One line was deliberately left out of that list. Same directory, same pattern, two shells:
# single file on disk, stored as NFD: çalışma.txt
bash pattern=NFC -> 0 matches pattern=NFD -> 1 match
zsh pattern=NFC -> 1 match pattern=NFD -> 0 matches
Exactly inverted. bash behaves as expected: it matches whatever bytes are on disk. zsh finds the NFD-stored name with an NFC pattern and fails with the NFD pattern. So zsh transforms the filename after reading it.
Rather than guess at the transformation, I went to the source. zsh's Src/utils.c holds the readdir() wrapper zreaddir(), and the comment above it says everything in one sentence: when __APPLE__ is defined, directory entry names are recoded from UTF-8-MAC to UTF-8. The code does precisely that:
#if defined(HAVE_ICONV) && defined(__APPLE__)
if (!conv_ds)
conv_ds = iconv_open("UTF-8", "UTF-8-MAC");
UTF-8-MAC is the name macOS's iconv gives to the decomposed form. That was worth checking rather than assuming:
$ printf 'a\xc4\x9f' | iconv -f UTF-8 -t UTF-8-MAC | od -An -tx1
61 67 cc 86 # NFC input -> decomposed
$ printf 'ag\xcc\x86' | iconv -f UTF-8-MAC -t UTF-8 | od -An -tx1
61 c4 9f # NFD input -> composed
So zsh converts every name it reads from a directory into composed form (NFC) and does all of its matching on that form. Most of the time this is exactly what you want — the form your keyboard produces is NFC, so Turkish filenames in zsh "just work". The price is that in a single zsh script, one file travels under two different strings.
g=(*) # the shell's own glob
s=($(/bin/ls)) # an external command's output
[[ ${g[1]} == ${s[1]} ]] && echo YES || echo NO
NO
glob raw: 61 c4 9f 2e 74 78 74 # NFC
ls raw: 61 67 cc 86 2e 74 78 74 # NFD
There is exactly one file in that directory. /bin/ls calls its own readdir, so it hands over the NFD bytes from disk; zsh's * passes through its wrapper, so it hands over NFC. Combine the two in one script and you land in a world where a file fails to equal itself. Before reading that source, three suspects were on my list — the shell, the file system and Python — and only one of them was involved.
The code carries one more branch: if the conversion errors out or finishes incomplete, zsh keeps the original name. On APFS you cannot trigger that branch, for a reason that comes up shortly.
In git, the form of the name is a setting's decision
The same story has a repository chapter, and there the choice of form is explicitly configurable. I put a file with an NFD name into an empty repository:
$ git config core.precomposeunicode
true
$ git add -A
$ git ls-files # path written into the index
U+0061 U+011F ... -> NFC
$ git -c core.precomposeunicode=false rm -r --cached -q .
$ git -c core.precomposeunicode=false add -A
$ git -c core.precomposeunicode=false ls-files
U+0061 U+0067 U+0306 ... -> NFD
One single file in the working tree; two different path names in the index. The git config documentation describes the setting as reverting "the unicode decomposition of filenames done by Mac OS", and states the purpose: sharing a repository with Linux or Windows. The default is true on this machine and true in the blog repository.
The practical consequence: a Turkish-named file committed from a Mac with false looks like "deleted plus newly added" on a Mac running with true. Two developers, one file, one NFC and one NFD — and git status is telling both of them the truth. Pinning the setting to a single value across a team removes a whole class of noisy diffs before they appear.
Looking at the blog repository through this lens produced a reassuring result: of its 6,635 tracked paths, zero are non-ASCII. The repository is immune, because none of the slugs carry Turkish characters.
Invalid UTF-8: APFS stops it at the door
On Linux a filename can be any byte sequence except / and NUL; there is no requirement that it be valid text. Not so on APFS. Apple's documentation is explicit: "APFS accepts only valid UTF-8 encoded filenames for creation." The same page names a second gate — APFS also refuses to create files whose names contain codepoints unassigned in Unicode 9.0, which HFS+ allowed.
Imitating a Turkish name encoded in ISO-8859-9, with a lone 0xF0 byte:
>>> open(b'a\xf0.txt', 'wb')
OSError: [Errno 92] Illegal byte sequence: b'a\xf0.txt'
The file is never created. So every filename sitting on an APFS volume is valid UTF-8 and can be normalized. Do not generalize that guarantee to the whole Mac: on an exFAT disk plugged into the same machine, or an SMB or NFS share, names follow that volume's rules and invalid UTF-8 can show up. That is also where zreaddir's "conversion failed, keep the original" branch becomes reachable.
The Linux counterpart of this topic came up in the article where I turned on casefold, and the two point in opposite directions. There the file system was deliberately made more tolerant, and cp, rsync and tar quietly ate a file — tolerance produced data loss. Here the file system is already tolerant and the losing side is the tools themselves: the file is right there, and whoever searches cannot find it.
Archives and sync: one file comes out short
This is where theory turns into practice. Inside a Linux container (overlay/ext4) I created two separate files whose names differ only in normalization form. On Linux they really are two files:
Linux file count: 2
U+0061 U+0067 U+0306 .txt -> 'nfd'
U+0061 U+011F .txt -> 'nfc'
Both names went into a single zip, which then travelled to the Mac. Reading the archive on the Mac confirmed two entries inside it. On extraction, the entry order inside the archive decides the outcome — which is why the results below are written as "first/last entry" rather than by content. I packed the zip twice, reversing the entry order the second time:
$ unzip iki-bicim.zip # no flags
replace ag??.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename:
(EOF or read error, treating as "[N]one" ...)
exit=1 -> 1 file, the FIRST entry's contents
$ unzip -o iki-bicim.zip
exit=0 -> 1 file, the LAST entry's contents
$ ditto -x -k iki-bicim.zip target
exit=0 -> 1 file, the LAST entry's contents
Flagless unzip plays it straight: it sees the collision, asks, gets no answer, stays with the first entry and returns a non-zero exit code. unzip -o writes the last entry over the first and returns zero. Apple's own ditto does the same — no warning, no question, clean exit. A two-entry archive yields one file, and nothing anywhere records it. Anyone running unzip -o in CI will never notice the loss.
On the sync side the common advice is rsync --iconv=UTF-8-MAC,UTF-8. It does not work on this machine:
$ rsync --iconv=UTF-8-MAC,UTF-8 t3/ t6/
rsync: unrecognized option `--iconv=UTF-8-MAC,UTF-8'
Because /usr/bin/rsync is now openrsync ("protocol version 29", introducing itself as "rsync version 2.6.9 compatible") with no --iconv support. The recipe you find online fails on this machine loudly rather than quietly. If you need the conversion, you have to install rsync separately or solve it at the archiving step instead.
How many names on my own machine are exposed
To keep this from staying abstract, I counted them. The script walks the home directory reading directory entries only, never opening a file:
import os, unicodedata as u
from collections import Counter
SKIP = {'Library', '.Trash', 'OneDrive', 'node_modules', '.git', '.npm', '.cache'}
kind, where = Counter(), Counter()
total = 0
def walk(path, depth=0):
global total
if depth > 8:
return
try:
entries = list(os.scandir(path))
except OSError:
return
for e in entries:
name = e.name
total += 1
if not name.isascii():
nfc, nfd = u.normalize('NFC', name), u.normalize('NFD', name)
if nfc == nfd: kind['undecomposable'] += 1
elif name == nfc: kind['NFC'] += 1
elif name == nfd:
kind['NFD'] += 1
where[os.path.relpath(os.path.join(path, name),
os.path.expanduser('~')).split('/')[0]] += 1
try:
if e.is_dir(follow_symlinks=False) and name not in SKIP and not name.startswith('.'):
walk(e.path, depth + 1)
except OSError:
pass
Two consecutive runs gave the same output:
entries scanned : 519,137
non-ASCII names : 398
stored as NFC : 274
stored as NFD : 59
undecomposable : 65
Fifty-nine names sit there in decomposed form; 8 are directories, 51 files. The spread: Desktop 31, Downloads 17, Music 3, Parallels applications 3, IdeaProjects 2, Documents 2, Pictures 1. Here is what matters: none of those 59 is reachable with an NFC pattern typed on my keyboard. Not find, not grep, not glob. As a percentage it is small, but there is a class your scanning script never sees, and you learn its size only by looking.
The non-ASCII characters appearing in those 59 names: ı 49, İ 36, ü 36, en dash – 19, ş 10, ğ 7, ö 6, the rest in single digits. ı and the en dash on that list are passengers, not culprits: neither decomposes, and what makes the name NFD is the ü, İ, ş, ğ or ö beside them. Fittingly, the scan turned up no NFD name without a decomposing letter in it.
Of the 65 "undecomposable" names, 60 contain only ı; the remaining 5 carry an em dash (—). So that class does not mean "just ı", it means "any non-ASCII character with no mark to shed". In practice both land in the same place: those names are identical in either form and therefore exempt.
One name deserves a special mention: Photo Booth Arşivi. The thing producing that decomposed name is not some zip I downloaded, it is Apple's own application. The comfortable explanation that "NFD is a leftover from old Macs" collapses on that one entry.
The recipe for getting it right
Everything above seems to me to collapse into a single rule: a name loses its identity the moment you turn it into text, so shorten the time it spends as text.
If you are asking whether something exists, do not list names — try to open it. os.path.exists and stat work regardless of normalization, because they hand the question to the kernel. To learn whether two paths point at the same file, reach for os.path.samefile instead of comparing names; it looks at the inode. It has one condition: both paths must exist, or it raises — and the case you are debugging is usually exactly the case where one path does not, so wrap it in a try.
If you must work with text — a search term from a user, a manifest file, a CSV — pin the form at the boundary. Convert every incoming name and every incoming pattern to the same form, then compare. In Python that is unicodedata.normalize("NFC", name), in Node name.normalize("NFC"). Which form you pick does not matter as long as you are consistent. A collision scan is built on the same logic:
import os, unicodedata, collections
def scan(root, form="NFC"):
groups = collections.defaultdict(list)
for folder, _, files in os.walk(root):
for name in files:
key = unicodedata.normalize(form, name)
groups[key].append(os.path.join(folder, name))
return {k: v for k, v in groups.items() if len(v) > 1}
A non-empty dictionary means that directory holds two names that look identical. On APFS this output should be empty. Run the same script on the Linux side before copying a directory to a Mac, and you see in advance which pair is about to merge. If you do not want them merged, rename before copying; afterwards is too late, because ditto will not tell you.
If you write shell scripts, do not mix the two routes. Combining * with $(ls) output in one script gives you two strings for one file under zsh. My preference is the glob: thanks to zsh's conversion it matches the form my keyboard produces, and $(ls) is already broken for names containing spaces.
Finally, about today's mess: renaming your existing NFD names to NFC looks tempting, but a rename is not a cheap operation. Symlinks, backup manifests, Time Machine history and hardcoded paths in your own scripts all keep pointing at the old name. My own approach would be to measure first, convert only the directories that are actively searched, and leave the archives alone.
Questions for your own setup
Four questions, each with a measurable answer:
- How many names in your home directory are stored decomposed? Run the scanning script above and get the NFC/NFD split. If the number is small, relax — but do notice that it is not zero.
- Does your repository contain non-ASCII paths? Filter
git ls-filesoutput throughisascii(). If any show up, makecore.precomposeUnicodeidentical across the whole team. - Does any automation of yours extract archives with
-o? If so, price out dropping that flag once; the cost is probably zero. - If you sync files between a Mac and Linux, do you run a collision scan on the Linux side before syncing?
Conclusion
The single image that stays with me from this is a directory holding one file, where that file fails to equal itself inside the same script. Nothing broke, no command errored, no line landed in any log. The same name simply became two different strings because it was read along two different routes.
That is why I do not find "normalize everything" to be sufficient advice. Normalization is a precaution; the real question is why the name became text at all. Most of the time the answer is that it did not have to — you could have opened the file directly instead of listing and filtering, you could have asked samefile instead of comparing names. The 59 names on my Mac are there to remind me of that; I am not going to fix them, because they are not the thing that needs fixing.
Top comments (0)