I Chased a Missing Fixture for 48 Hours. Linux Stored NFC. My Mac Stored NFD.

I spent a full forty-eight hours chasing a fixture that vanished only after I left my laptop. Have you ever watched pytest fail on a path that ls still prints without a single complaint? The JSON file lived in the repo, Git status stayed quiet, and my Mac kept loading data/café.json like the suite was healthy.

Then I ran the same command on a clean Linux box, and Python raised FileNotFoundError for a name I could still see. Why would two POSIX systems disagree about a filename that looks identical in a terminal font? I did not start with Unicode. I started with every local-versus-remote story that had saved me before, and each one wasted another hour.

What I blamed first

The failure looked like an environment problem, so I treated it like one. I have lost days to PYTHONPATH, pytest root detection, and locale, and those scars still steer my first guesses. Does that sound familiar, or do you jump straight to the filesystem bytes?

I wrote the wrong theories down so I would not keep rerunning them in a loop:

1. A polluted PYTHONPATH that only existed inside my laptop's interactive shell profile.
2. Pytest choosing a different rootdir once the working directory changed on the remote box.
3. A .dockerignore rule that dropped data/ when I copied the tree for a clean run.
4. A LANG mismatch, because I have already lost time to locale in other field notes.

Each theory was cheap to test and wrong in a slightly different way. The path existed in the listing. The ignore file was empty. Locale was C.UTF-8 on both sides after I forced it. Have you noticed how a green local suite makes you defend the laptop instead of distrusting it?

echo "PYTHONPATH=${PYTHONPATH-}"
python3 -c "import pathlib; print(pathlib.Path('data').resolve())"
pytest --collect-only -q tests/test_cafe_fixture.py
locale

Those commands all looked healthy. That was the problem. Healthy output is not the same thing as the same string.

Hour 8: ls lied, repr() did not

I stopped trusting the shell glyph and printed the directory with Python. The bytes were not friendly anymore, and the font stopped being evidence. Would you have reached for repr() this early, or would you have kept staring at ls?

from pathlib import Path

for path in sorted(Path("data").iterdir()):
    name = path.name
    print(name)
    print(repr(name))
    print([hex(ord(ch)) for ch in name])

On the Mac I saw 'café.json' with code points that included 0x65 and 0x301. On Linux I saw 'café.json' with a single 0xe9. Same glyph. Different string. Python equality does not care that your terminal folded them into one accented letter.

NFC stores é as U+00E9. NFD stores it as e plus combining acute U+0301. Path("data/café.json").exists() is a byte-level question on Linux, and a much softer question on a folding Mac volume. I had been debugging a font, not a path.

import unicodedata

nfc = unicodedata.normalize("NFC", "café.json")
nfd = unicodedata.normalize("NFD", "café.json")
print(nfc == nfd)  # False
print([hex(ord(c)) for c in nfc])
print([hex(ord(c)) for c in nfd])

That snippet is not a conference gotcha. It was the entire outage. My test helper built the NFC name because I typed the character in a Linux-flavored editor. My laptop created the file through a macOS save dialog that preferred NFD. Why did I trust a visual match after years of debugging hidden bytes?

Hour 20: Git stayed quiet, and that was the trap

I wanted Git to be the referee. git status was clean on both machines, so I assumed the blob matched. Did you know macOS Git often sets core.precomposeunicode, so the index can store NFC even when the working tree lists NFD?

git config --show-origin --get core.precomposeunicode
git ls-files -z data | xargs -0 python3 -c "
import sys, unicodedata
for raw in sys.stdin.buffer.read().split(b'\0'):
    if not raw:
        continue
    name = raw.decode()
    nfc = unicodedata.normalize('NFC', name)
    nfd = unicodedata.normalize('NFD', name)
    print(repr(name), name == nfc, name == nfd)
"

The index looked NFC. The working tree on my Mac still handed Python an NFD Path. Linux had no such courtesy. The checkout was "correct" and open() still failed, which is a sentence I had to write twice before I believed it. What broke my mental model was treating Git cleanliness as filesystem cleanliness.

Those are different layers. One stores objects. The other answers open(). git status can be green while your runtime still constructs a name the disk does not own. If your helper builds paths from literals in source, Git will not save you.

Hour 32: I needed a disk that would not fold names

I could not keep using my laptop as the source of truth. macOS was being helpful, and helpful filesystems hide collisions. I copied the repo onto a Linux box and ran the same listing script with no extra packages. That is when the second encoding stopped being theoretical.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I already had MonkeyCode's free server option available, so I used that box as a second filesystem rather than as a product demo. I also used the free model access to draft a first pass of a directory auditor, then I rewrote the comparison logic by hand until repr() and unicodedata agreed. The model did not magically know my tree. It did save me from starting the scanner from a blank buffer at hour thirty-two.

Would I ship the first draft it produced? No. The first version normalized in memory and then called exists(), which is exactly the bug on a folding filesystem. I made the auditor print raw code points and refuse to treat a successful exists() as proof. Help that papers over the disk is not help.

The artifact: a normalization audit you can run

This is the workflow I would repeat. It is deliberately boring. It does not need special hardware, and it does not care which editor created the file. Label the commands as a lab you can run; they are the reproduction, not a claim about production metrics.

Decision table I wish I had on hour one

• If two encodings print the same glyph, a folding Mac volume may keep one directory entry.
• If two encodings print the same glyph, a typical Linux ext4 volume may keep two directory entries.
• If path.exists() succeeds after you normalize only in memory, you may still be on a folding filesystem.
• If git status is clean, you have index health, not open() health.
• If you only inspect names with ls, you are debugging a font.

Question	Folding Mac volume	Typical Linux ext4
Write NFD, then exists(NFC)	often true	true only if NFC exists
Two encodings, one glyph	one entry is common	two entries are possible
Trust ls	no	no
Trust repr(name)	yes	yes
Trust git status alone	no	no

Auditor script

#!/usr/bin/env python3
"""Audit a tree for Unicode filename normalization mismatches.

Label: local diagnostic, not a production crawler.
"""
from __future__ import annotations

import argparse
import json
import sys
import unicodedata
from pathlib import Path

SKIP_PARTS = {".git", "__pycache__", ".venv", "node_modules"}


def forms(name: str) -> dict[str, str]:
    return {
        "raw": name,
        "nfc": unicodedata.normalize("NFC", name),
        "nfd": unicodedata.normalize("NFD", name),
        "codepoints": " ".join(f"U+{ord(ch):04X}" for ch in name),
    }


def audit(root: Path) -> list[dict]:
    findings: list[dict] = []
    seen: dict[str, list[str]] = {}
    for path in root.rglob("*"):
        if any(part in SKIP_PARTS for part in path.parts):
            continue
        record = forms(path.name)
        nfc_key = str(path.parent / record["nfc"])
        seen.setdefault(nfc_key, []).append(str(path))
        if record["raw"] != record["nfc"] or record["raw"] != record["nfd"]:
            findings.append(
                {
                    "path": str(path),
                    "issue": "non_ascii_name",
                    **record,
                    "nfc_equals_raw": record["raw"] == record["nfc"],
                    "nfd_equals_raw": record["raw"] == record["nfd"],
                }
            )
    for key, paths in seen.items():
        unique = sorted(set(paths))
        if len(unique) > 1:
            findings.append({"path": key, "issue": "nfc_collision", "aliases": unique})
    return findings


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("root", type=Path, nargs="?", default=Path("."))
    args = parser.parse_args()
    rows = audit(args.root)
    json.dump(rows, sys.stdout, indent=2, ensure_ascii=False)
    sys.stdout.write("\n")
    return 1 if rows else 0


if __name__ == "__main__":
    raise SystemExit(main())

Pair it with a test that fails when the checked-in tree still carries a collision. The test does not try to be clever. It only refuses a dirty audit of data/.

# tests/test_filename_forms.py
import json
import subprocess
import sys
from pathlib import Path

def test_audit_is_clean_on_checked_in_tree():
    repo = Path(__file__).resolve().parents[1]
    proc = subprocess.run(
        [sys.executable, str(repo / "scripts" / "filename_audit.py"), str(repo / "data")],
        check=False,
        capture_output=True,
        text=True,
    )
    rows = json.loads(proc.stdout or "[]")
    collisions = [row for row in rows if row.get("issue") == "nfc_collision"]
    assert collisions == [], collisions

Repro lab

How do you seed a reproduction without wrecking a real repo? Create both forms under a throwaway directory and compare what the disk kept.

python3 - <<'PY'
import unicodedata
from pathlib import Path
root = Path("/tmp/nfc-nfd-lab")
root.mkdir(exist_ok=True)
for child in root.iterdir():
    child.unlink()
nfc = unicodedata.normalize("NFC", "café.json")
nfd = unicodedata.normalize("NFD", "café.json")
(root / nfc).write_text('{"form":"nfc"}', encoding="utf-8")
try:
    (root / nfd).write_text('{"form":"nfd"}', encoding="utf-8")
except FileExistsError:
    print("filesystem folded the second write")
print("listed:", [repr(p.name) for p in root.iterdir()])
PY
python3 scripts/filename_audit.py /tmp/nfc-nfd-lab

On a folding Mac volume, the second write may overwrite the first, and the listing still looks like one file. On a typical Linux ext4 volume, you can get two directory entries that print the same glyph. That is the whole point of keeping a Linux disk in the loop. Have you ever watched a "duplicate" file appear only after you left APFS behind?

What I would repeat next time

I would not start with Docker, pytest plugins, or another locale rabbit hole. I would force repr() on every path the test constructs, then I would run the same listing on a Linux disk that does not fold names. The forty-eight hours were not a mystery of pytest. They were a mystery of me trusting a glyph.

• Print repr(path) and code points before you print a human-readable name.
• Compare NFC and NFD explicitly; never trust font rendering in a terminal.
• Treat git status as index health, not as open() health.
• Keep one case-sensitive, non-folding filesystem in the debug loop.
• Refuse any helper that calls exists() after normalizing only in memory.

Would I still use an editor on macOS? Yes. I would just stop treating that laptop as the filesystem that CI will see. The next fixture with a non-ASCII name gets the auditor before it gets a shrug.

Limitations, and who should not bother

This workflow does not fix Windows short names, APFS case folding, or SMB servers that rewrite names on copy. It also does not prove that two files are semantically the same document. It only tells you whether Python will see one path or two. If your code never hits disk, a Linux reproduction box will not save you.

Skip this if your tree is ASCII-only and already built on Linux CI for every pull request. Skip it if you cannot copy fixtures without secrets, because a remote box is still a machine you do not fully control. Skip it if you need an air-gapped model or dedicated hardware; I am only talking about free model access and a second filesystem, not a production cluster.

Normalize at the boundary where you accept names, then store one form on purpose. If you already have a spare Linux machine, use that and keep the scanner in the repo. If you do not, I used MonkeyCode's free server option for this audit and still treated the chat draft as disposable.