My Laptop Spoke UTF-8. The Clean Process Spoke ASCII.

I spent the first pass blaming the parser, which feels familiar, and it was the wrong suspect again. The sample was a few UTF-8 bytes plus a line ending, and the same function read it cleanly on my laptop. Why did a clean process, handed that same file, raise UnicodeDecodeError before any parser code had a chance to run? I am writing these notes so I do not land another text-mode read that only works because my shell is polite.

What I thought had broken

I assumed the CSV library had changed, because the traceback sat one frame above the read. Had I upgraded something between yesterday and this morning, or was I staring at an old virtualenv again? I held the interpreter path constant on purpose, since a different sys.executable is a separate note I already published. This time the binary matched, the file bytes matched, and the failure still picked a side.

The next wrong guess was a hidden dependency, so I froze the environment and ran the snippet with nothing else imported. The failure survived that cut, which should have ended the package theory on the spot. I then compared newlines, permissions, and the working directory, and none of those knobs moved the exception. What else was the process inheriting that never shows up in a diff of the source file?

The fact I had blurred

Calling sys.getdefaultencoding() has returned utf-8 for years, and that return value still does not govern open(). Text mode follows the locale encoding unless UTF-8 mode is already on, which you can see in sys.flags.utf8_mode. So a laptop with LANG=C.UTF-8 and a process with LC_ALL=C are not running the same read, even when the snippet looks identical. Are you checking the encoding the file actually used, or the encoding you remember from a tutorial?

Python 3.10 and later can warn on this if you set PYTHONWARNDEFAULTENCODING=1, which is the EncodingWarning from PEP 597, not a change of default. I am not going to claim that a later release flipped that default, because a version rumor is how this class of bug hides. If UTF-8 mode is on, the flag says so, and I trust that flag over any blog date. locale.getencoding() needs 3.11 or newer, so the probe below falls back instead of pretending every host has it.

The probe I would repeat

I wrote a small probe that writes a known UTF-8 sample and then reads it back with no encoding argument. That omission is the bug I want to catch, so the probe fails in the same way a careless helper fails. It also prints the locale variables, the preferred encoding, and the UTF-8 mode flag, because a green read without that context is useless. Would you trust a passing test that never recorded the locale it actually ran under?

"""Bare open() probe. Expected behavior is labeled; this is not a benchmark."""
import json
import locale
import os
import sys
import tempfile
from pathlib import Path

def snapshot():
    info = {
        "executable": sys.executable,
        "version": sys.version.split()[0],
        "default_encoding": sys.getdefaultencoding(),
        "filesystem_encoding": sys.getfilesystemencoding(),
        "preferred_encoding": locale.getpreferredencoding(False),
        "utf8_mode": bool(sys.flags.utf8_mode),
        "lang": os.environ.get("LANG"),
        "lc_all": os.environ.get("LC_ALL"),
        "pythonutf8": os.environ.get("PYTHONUTF8"),
        "pythonioencoding": os.environ.get("PYTHONIOENCODING"),
    }
    if hasattr(locale, "getencoding"):
        info["locale_getencoding"] = locale.getencoding()
    return info

def probe(directory: Path) -> dict:
    path = directory / "sample.txt"
    path.write_bytes("café\n".encode("utf-8"))
    result = {
        "snapshot": snapshot(),
        "read_ok": None,
        "error": None,
        "round_trip": None,
    }
    try:
        # Intentional: no encoding argument. This is the bug under test.
        text = path.read_text()
        result["read_ok"] = True
        result["round_trip"] = text == "café\n"
    except UnicodeDecodeError as exc:
        result["read_ok"] = False
        result["error"] = f"{type(exc).__name__}: {exc}"
    return result

def main():
    with tempfile.TemporaryDirectory() as raw:
        print(json.dumps(probe(Path(raw)), indent=2))

if __name__ == "__main__":
    main()

I save that as locale_open_probe.py and run four commands, changing the environment rather than the source. The first command is my laptop as I actually work. The next two force a UTF-8 locale and a C locale. The last one keeps the C locale and turns UTF-8 mode on, so I can see which switch really healed the read.

python3 locale_open_probe.py
LC_ALL=C.UTF-8 LANG=C.UTF-8 python3 locale_open_probe.py
LC_ALL=C LANG=C python3 locale_open_probe.py
LC_ALL=C LANG=C PYTHONUTF8=1 python3 locale_open_probe.py
PYTHONWARNDEFAULTENCODING=1 python3 -c 'open("sample.txt")'

What broke when only the locale moved

On a glibc Linux user, with UTF-8 mode off, I expect LC_ALL=C to report an ASCII locale such as ANSI_X3.4-1968. I also expect the bare read to raise, because the byte 0xC3 in café is not ASCII. That expectation comes from the locale-dependent default documented for open(), not from a timer and not from a scoreboard. If your C locale still round-trips the file, the test did not fire, so read preferred_encoding before you celebrate a false green.

A second mask showed up when I set PYTHONIOENCODING=utf-8 and left open() alone. Standard streams looked healthy, and the file read still died, which is an easy way to blame the wrong layer. Why would a pipe setting repair a filesystem read that never consulted that variable? I would repeat the probe with that variable both set and unset, and I would keep the two results in the same note.

The fix I would land for files this repo owns is explicit, including newline policy when the bytes matter:

text = path.read_text(encoding="utf-8", newline="")
path.write_text(text, encoding="utf-8", newline="\n")

A second host is not a second opinion

When I want a clean process without building another VM, I borrow MonkeyCode's free server option and run this same probe there. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I do not know that server's image, locale, hardware, quota, or how long the free option lasts, so I will not invent those facts in this note. The useful part is a process whose environment I did not inherit from my laptop shell.

I copy the probe, run the four commands, and diff the JSON keys rather than the prose around them. The keys I actually compare are preferred_encoding, utf8_mode, read_ok, and round_trip. Free model access helps one step later, when I ask for a patch that pins encoding="utf-8" and names the C-locale command. I still drop that patch if it explains the error without a command I can re-run on both hosts.

Decision table I keep next to the probe

What the file is	Encoding I pass	Newline	Why I choose that
Data files this repo owns	utf-8	"" when bytes matter	The locale must not rewrite them
A legacy export already in a code page	The measured locale, named in the log	Default only if I mean translation	Pinning UTF-8 would corrupt it
A human terminal or a pipe	Leave it, or set PYTHONIOENCODING knowingly	Not applicable	Forcing UTF-8 can break an ASCII pipe
The test job	utf-8, plus a C-locale run of the probe	""	The C job is what catches a bare open()

What I would repeat before I land the edit

1. I print the snapshot before I read the traceback, so the locale is evidence instead of a guess.
2. I run the C locale and the UTF-8 locale back to back, and I keep both JSON blobs in the note.
3. I treat a model draft as a suggestion, then I re-run the probe on the laptop and on the second host.
4. I pin encoding="utf-8" for repo-owned text, and I leave a comment when a file is intentionally locale-encoded.
5. I check PYTHONIOENCODING separately, because a healthy pipe can hide a broken open().
6. I refuse to call the check done if preferred_encoding never left UTF-8, since the ASCII case never ran.

Who should skip this recipe

You should skip the forced C locale if the files are genuine user documents in a legacy code page, because a UTF-8 pin will damage them. You should also skip a random clean host when production uses a measured locale you have not copied, since a false green is worse than no second run. Windows hosts do not honor LC_ALL=C the way glibc does, so this shell recipe alone will not probe an ANSI code page. Teams that already pass an encoding on every text read still need the warning flag, but they do not need this story.

This note will not catch a CSV dialect bug, a delimiter change, or a hash mismatch in a deployed tree. It will not tell you which model wrote a better patch, and it is not a performance result. macOS builds sometimes keep UTF-8 even under a C locale, and the probe is inconclusive there until the printed encoding actually changes. If round_trip stays true everywhere, I would stop and look at a different layer rather than tighten a rule that never fired.

If you do not already have a spare Linux user where you can flip LC_ALL, MonkeyCode's free server is the second process I would borrow for this probe, and I would still read the JSON before I edited a single call.