Two Files Named café.mp4 in One Folder: How Unicode Normalization Defeats a Copy Script's Skip Check

A small Python script copies new clips from a phone folder into an archive and skips anything already listed in archived.txt. The list says café.mp4 is done. The script copied café.mp4 anyway, and now ls prints the same name twice in one folder. Neither file is a duplicate in the way the script understands duplicates: the two names are different byte strings that render as the same word.

Disclosure: I make an Android camera app. Nothing below uses its code or describes how it names or stores files. Everything here ran on one Linux machine (ext4, kernel 6.8, Python 3.10.12) against files the scripts create themselves.

What the tests showed:

• é can be one code point or two. The composed form is U+00E9. The decomposed form is a plain e followed by U+0301 COMBINING ACUTE ACCENT. They print identically.
• The folder kept both names. This ext4 folder treated the two spellings as two different files.
• A plain in check misses the match. Python string equality compares code points, so the skip list did not match the clip on the phone.
• A normalized comparison key fixes the skip check, and a collision check catches a folder holding both spellings.
• Renaming to the composed form can destroy a clip. os.rename replaced the other file silently.

The setup

setup.py writes one clip to phone/ under the decomposed name, writes archived.txt with the composed name as if an earlier run had recorded it, and prints both spellings.

import os, unicodedata

nfc = unicodedata.normalize("NFC", "café.mp4")
nfd = unicodedata.normalize("NFD", "café.mp4")

os.makedirs("phone", exist_ok=True)
os.makedirs("archive", exist_ok=True)

# The phone's copy of the clip is stored with a decomposed name.
with open(os.path.join("phone", nfd), "wb") as f:
    f.write(b"clip bytes\n")

# The archive list was written by a script that used the composed form.
with open("archived.txt", "w", encoding="utf-8") as f:
    f.write(nfc + "\n")

for label, s in (("composed  ", nfc), ("decomposed", nfd)):
    print(label, len(s), "code points", s.encode("utf-8"))

composed   8 code points b'caf\xc3\xa9.mp4'
decomposed 9 code points b'cafe\xcc\x81.mp4'

Eight code points against nine, and the bytes differ in the middle: \xc3\xa9 is the UTF-8 encoding of the single composed character, while e\xcc\x81 is a plain e followed by the two bytes of the combining accent.

The Python Unicode HOWTO describes the same kind of pair, using ê (U+00EA, or e followed by U+0302): "These will produce the same output when printed, but one is a string of length 1 and the other is of length 2."

seed_archive.py puts the earlier copy into archive/ under the composed name, matching the list.

import os, unicodedata

# The archive already holds the clip under its composed name from an earlier run.
nfc = unicodedata.normalize("NFC", "café.mp4")
with open(os.path.join("archive", nfc), "wb") as f:
    f.write(b"clip bytes\n")

The skip check that does not skip

This is the naive version. It reads the list into a set and skips any phone file whose name is in it.

import os, shutil

with open("archived.txt", encoding="utf-8") as f:
    done = set(line.rstrip("\n") for line in f)

for name in sorted(os.listdir("phone")):
    if name in done:
        print("skip  ", name)
        continue
    shutil.copy2(os.path.join("phone", name), os.path.join("archive", name))
    print("copied", name)

copied café.mp4

It copied a clip the list says is already archived. ls -1 archive prints café.mp4 twice; here is each name's length and bytes.

'café.mp4' 9 b'cafe\xcc\x81.mp4'
'café.mp4' 8 b'caf\xc3\xa9.mp4'

Two entries, same visible name, different lengths, different bytes. This ext4 folder stored each name as the bytes it was given and did not treat the two spellings as the same name, so both files exist side by side. Python's name in done compares code points, and the composed and decomposed strings are different sequences of code points, so the membership test returned False and the copy ran.

The unicodedata documentation says it plainly: "Even if two unicode strings are normalized and look the same to a human reader, if one has combining characters and the other doesn’t, they may not compare equal."

Where the decomposed spelling comes from, I cannot tell you. A name carries whatever code points the program that created it emitted, and I did not test any device or operating system. Assume a folder fed from more than one source can contain both spellings.

Comparing on a normalized key

Unicode defines normalization forms precisely so that equivalent strings can be compared. From UAX #15: "Two equivalent strings will have precisely the same normalized form." In particular, if two strings are canonical equivalents, then toNFC(x) = toNFC(y).

NFC is the composed form. In the unicodedata docs, "Normal form C (NFC) first applies a canonical decomposition, then composes pre-combined characters again." The fixed script normalizes both sides, list and phone, to NFC before comparing, and copies under the NFC name so the archive stops collecting both spellings.

import os, shutil, unicodedata

def key(name):
    return unicodedata.normalize("NFC", name)

with open("archived.txt", encoding="utf-8") as f:
    done = set(key(line.rstrip("\n")) for line in f)

seen = {}
for name in sorted(os.listdir("phone")):
    k = key(name)
    if k in seen:
        print("COLLISION", repr(seen[k]), "and", repr(name), "-> same key, not copying")
        continue
    seen[k] = name
    if k in done:
        print("skip  ", name)
        continue
    shutil.copy2(os.path.join("phone", name), os.path.join("archive", k))
    print("copied", name, "as", repr(k))

Run against a fresh archive that holds only the composed copy:

skip   café.mp4

'café.mp4' 8 b'caf\xc3\xa9.mp4'

One file, the composed name, and the skip worked.

The seen dictionary matters for the case where the phone folder itself already holds both spellings. both_on_phone.py adds a second copy under the composed name.

import os, unicodedata

# A second copy of the clip lands on the phone under the composed name.
nfc = unicodedata.normalize("NFC", "café.mp4")
with open(os.path.join("phone", nfc), "wb") as f:
    f.write(b"clip bytes\n")

skip   café.mp4
COLLISION 'café.mp4' and 'café.mp4' -> same key, not copying

The decomposed name sorts first and hits the skip list; the composed name then maps to a key that has already been seen, so the script reports it instead of copying it. It stops and reports rather than picking a winner, because it cannot know whether the two files hold the same clip. That is a decision to make on purpose, not by sort order.

The rename that ate a clip

The tempting cleanup is to rename every name to its NFC form. rename_trap.py writes an older clip under the composed name and a newer one under the decomposed name, then renames the decomposed one.

import os, unicodedata

os.makedirs("trap", exist_ok=True)
nfc = unicodedata.normalize("NFC", "café.mp4")
nfd = unicodedata.normalize("NFD", "café.mp4")
with open(os.path.join("trap", nfc), "w") as f:
    f.write("older clip\n")
with open(os.path.join("trap", nfd), "w") as f:
    f.write("newer clip\n")

# "Tidy" the decomposed name into the composed one.
os.rename(os.path.join("trap", nfd), os.path.join("trap", nfc))

for name in os.listdir("trap"):
    with open(os.path.join("trap", name)) as f:
        print(repr(name), "->", f.read().strip())

'café.mp4' -> newer clip

One file left, holding the newer content. The older clip is gone, with no exception and no warning. That is documented behavior. The pathlib documentation for Path.rename says: "On Unix, if target exists and is a file, it will be replaced silently if the user has permission." os.rename did the same thing in this test.

A safer rename refuses to overwrite. os.link creates a second name for the file and raises FileExistsError if the target name is taken, so the original name is only removed after the new one exists.

import os, unicodedata

def to_nfc(folder, name):
    new = unicodedata.normalize("NFC", name)
    if new == name:
        return "already NFC"
    src, dst = os.path.join(folder, name), os.path.join(folder, new)
    try:
        os.link(src, dst)          # fails if dst exists, unlike os.rename
    except FileExistsError:
        return "kept both: %r already exists" % new
    os.unlink(src)
    return "renamed"

os.makedirs("trap2", exist_ok=True)
nfc = unicodedata.normalize("NFC", "café.mp4")
nfd = unicodedata.normalize("NFD", "café.mp4")
for n, text in ((nfc, "older clip\n"), (nfd, "newer clip\n")):
    with open(os.path.join("trap2", n), "w") as f:
        f.write(text)

print(to_nfc("trap2", nfd))
print(sorted(os.listdir("trap2")))

kept both: 'café.mp4' already exists
['café.mp4', 'café.mp4']

Both files survived, and the function says why it did not rename. Which one to keep is, again, your call.

Hard links need a filesystem that supports them; on one that does not, a check-then-rename leaves a small race window.

NFC, not NFKC

Python offers four forms: "Valid values for form are ‘NFC’, ‘NFKC’, ‘NFD’, and ‘NFKD’." The K forms also apply compatibility mappings, and for filenames that folds characters that someone may have chosen on purpose.

import unicodedata

for name in ("clip①.mp4", "clip1.mp4", "fish-pond.mp4", "fish-pond.mp4"):
    print(repr(name), "NFC:", repr(unicodedata.normalize("NFC", name)),
          "NFKC:", repr(unicodedata.normalize("NFKC", name)))

'clip①.mp4' NFC: 'clip①.mp4' NFKC: 'clip1.mp4'
'clip1.mp4' NFC: 'clip1.mp4' NFKC: 'clip1.mp4'
'fish-pond.mp4' NFC: 'fish-pond.mp4' NFKC: 'fish-pond.mp4'
'fish-pond.mp4' NFC: 'fish-pond.mp4' NFKC: 'fish-pond.mp4'

Under NFKC, a circled digit becomes a plain digit and the fi ligature becomes two letters, so names that were distinct under NFC now share a key. UAX #15 describes NFC as the form that "maintains the distinction between characters that are compatibility equivalents." For a key whose only job is to say whether two names are the same name, that is the property you want. NFD would work equally well as a comparison key; I used NFC so the archive gets the composed form, which is the shorter of the two.

Normalization does not fold case: Café.mp4 and café.mp4 stay distinct under every form. For that, the Unicode HOWTO pairs casefold() with normalization.

Finding names that need attention

To check whether a folder already has the problem, this script walks a tree, prints every name that is not in NFC, and reports any directory that holds two names with the same NFC key. It prints raw bytes, because printed text is exactly what hides the difference. unicodedata.is_normalized was "Added in version 3.8."

import os, sys, unicodedata

root = sys.argv[1] if len(sys.argv) > 1 else "."
keys = {}
for dirpath, dirnames, filenames in os.walk(root):
    for name in dirnames + filenames:
        k = (dirpath, unicodedata.normalize("NFC", name))
        keys.setdefault(k, []).append(name)
        if not unicodedata.is_normalized("NFC", name):
            print("not NFC:", os.path.join(dirpath, name).encode("utf-8"))
for (dirpath, k), names in sorted(keys.items()):
    if len(names) > 1:
        print("same name twice in", dirpath + ":", [n.encode("utf-8") for n in names])

Run against the archive from the naive sync:

not NFC: b'archive/cafe\xcc\x81.mp4'
same name twice in archive: [b'caf\xc3\xa9.mp4', b'cafe\xcc\x81.mp4']

A non-NFC name is not an error by itself. It causes trouble when compared with a name from another source, or when its composed twin appears beside it.

What to change in a copy job

1. Decide on one normalization form for comparison keys. NFC is the usual choice.
2. Normalize both sides of every name comparison: lists, databases, manifests and directory listings.
3. When two names in one folder map to the same key, stop and report. Do not let sort order pick the survivor.
4. Do not rename with os.rename or Path.rename when the target might exist. Use a method that fails on an existing target.
5. Log names as bytes or repr(), not as printed text.

Things I did not test

• Other filesystems and options. Everything ran on one ext4 filesystem under Linux, in directories without the case-insensitive flag. Other filesystems, and filesystem options such as case-insensitive directories, may normalize or compare names differently, and I did not check any of them.
• Phones and file managers. I did not test which spelling any device, app or transfer tool produces.
• Other languages. Only Python's string comparison was tested. Other languages and shells may compare names differently.

Sources. Quotes are from the Python documentation for the unicodedata module, the Unicode HOWTO and pathlib's Path.rename, and from UAX #15: Unicode Normalization Forms. Test results come from the scripts above, run once each in the order shown in one folder (the archive was emptied and reseeded before the fixed sync), with the output pasted unedited.

If you want an old phone to record clips in the background so there is a folder worth copying, the app I make is Background Camera RemoteStream on Google Play. Nothing above depends on it.

Two spellings of one name are still one name; make the script agree before it copies or renames anything.