I Trusted open() for 48 Hours. The Remote Locale Still Spoke ASCII.

Have you ever shipped a parser that passed every local test, then exploded on a box you barely logged into? I spent a messy stretch chasing UnicodeDecodeError from a CSV that looked perfectly boring on my laptop. The file had a handful of city names with accents, nothing exotic, and my tests kept smiling at me. Why did a second machine treat those exact same UTF-8 bytes as an unreadable crime scene?

Field notes, hour 0–8: a green laptop and a confident draft

I started the way a lot of us start now, by asking an assistant to sketch a small reader. Disclosure: This article was prepared as part of MonkeyCode's product outreach, not as an independent lab report. I used its free model access to draft the parser, then I needed a second machine that was not my laptop. The free server option became that extra box, and the draft treated encoding like a solved problem.

Locally that draft was enough, because my desktop locale already spoke UTF-8 every morning. Names like Sao Paulo still needed real accents in the file, and those accents sailed through without a traceback. Did I print locale.getpreferredencoding() before I trusted that smiling green bar on my laptop? I did not print it, and that quiet omission ate the rest of my weekend.

# Labeled example: the first draft I almost kept.
import csv
from pathlib import Path


def load_cities(path: Path) -> list[dict[str, str]]:
    with path.open() as handle:  # encoding left to the locale
        return list(csv.DictReader(handle))

What the draft never asked me

Have you noticed how assistants echo the happy path you already live on every day? The sketch never set encoding="utf-8", never exported PYTHONUTF8, and never mentioned the PEP 597 default-encoding warnings. It also never asked whether the remote image shipped C.UTF-8, en_US.UTF-8, or a brutal LANG=C. I treated that silence as proof, which is a habit I am actively trying to break.

Python still takes the locale encoding for text mode on many 3.12 and 3.14 runtimes when UTF-8 mode is off. PEP 686 wants UTF-8 mode as the default, and Python 3.15 is aimed at October 2026. That is not a fleet you can assume on 19 September 2026, no matter how loud the release calendar feels. Until that default actually lands in the interpreters you ship, open(path) remains a portability bug in costume.

Hour 8–24: the second machine disagreed

I copied the CSV and the function onto the remote shell and ran the same pytest command I used at home. The first failure was not a logic error; it was UnicodeDecodeError about an ascii codec and byte 0xc3. Do you know that feeling when the stack points at a builtin and your brain still blames the file? I opened the CSV in less, saw the accents, and wasted an hour suspecting a corrupt copy.

Then I asked the remote interpreter a question I should have asked at hour one.

python -c "import locale, sys; print(sys.version); print(locale.getpreferredencoding(False)); print(sys.flags.utf8_mode)"
locale
echo "LANG=${LANG-} LC_ALL=${LC_ALL-}"

The laptop printed UTF-8 with utf8_mode=0, and it still decoded the file by luck. The remote printout said ANSI_X3.4-1968 after I reproduced the poor locale with LANG=C LC_ALL=C. Same script, same file, different preferred encoding, and the draft was correct only for my sofa.

I will not pretend a free hosted shell arrives magically with LANG=C already exported. I exported that locale on purpose after the first confusing failure, as a fixture I could rerun. The useful part was a second Python, a second libc, and environment variables I had not customized for comfort.

A probe you can run without my CSV

I got tired of arguing from screenshots, so I wrote a probe that fails in ASCII locales. Save this as probe_open_encoding.py, then run it in your normal shell and under LANG=C. It writes UTF-8 bytes, reads with the default encoding, and prints the preferred codec before it dies. If your desktop is already UTF-8, the second command is the one that tells the truth.

"""Repro: locale-dependent text mode. Run under LANG=C and under UTF-8."""
from __future__ import annotations

import locale
import os
import sys
import tempfile
from pathlib import Path

SAMPLE = "city,country\nZuerich-but-utf8,Switzerland\n"  # real file uses Zürich


def preferred() -> str:
    return locale.getpreferredencoding(False)


def write_utf8(path: Path) -> None:
    # Non-ASCII bytes that UTF-8 encodes as 0xc3 0xbc for ü.
    path.write_bytes("city,country\nZürich,Switzerland\n".encode("utf-8"))


def read_default(path: Path) -> str:
    # Intentionally unpinned, which is the bug under discussion.
    return path.read_text()


def read_pinned(path: Path) -> str:
    return path.read_text(encoding="utf-8")


def main() -> int:
    print(f"python={sys.version.split()[0]}")
    print(f"utf8_mode={sys.flags.utf8_mode}")
    print(f"preferred={preferred()!r}")
    print(f"LANG={os.environ.get('LANG')!r} LC_ALL={os.environ.get('LC_ALL')!r}")

    with tempfile.TemporaryDirectory() as raw:
        path = Path(raw) / "cities.csv"
        write_utf8(path)
        pinned = read_pinned(path)
        assert "Zürich" in pinned
        try:
            default = read_default(path)
        except UnicodeDecodeError as exc:
            print(f"default_open=FAILED {exc}")
            return 1
        print("default_open=OK")
        print(default.splitlines()[1])
        return 0


if __name__ == "__main__":
    raise SystemExit(main())

Then I wrapped the same facts in a pytest module that makes the locale matrix visible.

# test_open_encoding.py
import sys
from pathlib import Path

import pytest


def locale_pref() -> str:
    import locale

    return locale.getpreferredencoding(False)


def test_unpinned_read_depends_on_locale(tmp_path: Path) -> None:
    path = tmp_path / "row.txt"
    path.write_bytes("Zürich\n".encode("utf-8"))
    preferred = locale_pref()
    normalized = preferred.lower().replace("-", "")
    if sys.flags.utf8_mode or normalized in {"utf8", "utf8"}:
        assert "ü" in path.read_text()
        return
    with pytest.raises(UnicodeDecodeError):
        path.read_text()


def test_pinned_read_is_stable(tmp_path: Path) -> None:
    path = tmp_path / "row.txt"
    path.write_bytes("São Paulo\n".encode("utf-8"))
    assert "São" in path.read_text(encoding="utf-8")

Commands I now keep in the incident notes

Commands I now keep in the incident notes look like this painfully short block of shell.

python probe_open_encoding.py
LANG=C LC_ALL=C python probe_open_encoding.py
PYTHONUTF8=1 LANG=C LC_ALL=C python probe_open_encoding.py
PYTHONWARNDEFAULTENCODING=1 python -W default probe_open_encoding.py
pytest test_open_encoding.py -q

The first line is a comfort test. The second line is the actual test. The third line asks whether UTF-8 mode can paper over a missing encoding= argument. The fourth line makes Python hiss when generated code leaves the codec implicit.

The decision table I wish I had on hour one

Would a table have saved me the hour I spent hexdumping a perfectly valid UTF-8 file?

Situation	What I saw	What I do now
Desktop UTF-8 locale, utf8_mode=0	Unpinned open() passes	Still pin encoding="utf-8" for text I control
LANG=C or LC_ALL=C	UnicodeDecodeError on non-ASCII	Reproduce with the probe before blaming the file
PYTHONUTF8=1	Text mode behaves like UTF-8	Use it as a belt, not as the only belt
Incoming bytes from a vendor	Maybe not UTF-8 at all	Detect or negotiate; do not assume my laptop
Binary or latin-1 archives	Random decode success	Do not apply this CSV lesson blindly

I keep that table above the probe in the repo, because future me will forget the LC_ALL override again. A green laptop is not a locale matrix, and a locale matrix is not a vendor decoder. The table exists to stop me from mixing those three jobs into one panicked pytest run.

What broke when I "fixed" it the wrong way

I tried three bad fixes before I accepted the boring one that survived review.

1. I forced LANG=en_US.UTF-8 in my shell profile and declared the remote box permanently healed.
2. Wrapping values in encode().decode() laundered the strings without teaching me which codec had failed.
3. Asking the assistant to make it robust produced errors="ignore", which drops umlauts and keeps pytest green.

The profile trick hid the bug for my interactive user and left cron, systemd, and CI in the old locale. The encode-decode shuffle compiled, which is the worst kind of failure, because it looks like understanding. The errors="ignore" suggestion was the one that finally made me angry, because a missing city letter is worse than a loud traceback.

The fix that survived review was unglamorous, and I still like it more than the clever ones. I pinned encoding="utf-8" at every text boundary I owned and ran the probe under LANG=C. I turned on PYTHONWARNDEFAULTENCODING in the development tox factor so silent defaults hiss at me. I also stopped treating a local pytest pass as evidence about another machine's libc.

What I would repeat next time

• Print sys.flags.utf8_mode and locale.getpreferredencoding(False) in the first failing job, before I rewrite the parser.
• Ask the assistant for a locale matrix, not for a function that looks clean on one laptop.
• Run the probe under LANG=C on any spare Unix I can borrow, including a second remote shell when I do not have another box handy.
• Keep errors="strict" until a human names a vendor encoding in writing.
• Treat missing encoding= in generated code as a defect, the same way I treat a missing timeout.

Would I still use a generated first draft? Yes, because typing csv.DictReader is not the hard part. The hard part is noticing which defaults the draft inherited from my comfortable locale. Generated code is a sketch of the happy path, and the happy path is almost never the production path.

Who should not copy this workflow

If you already pin encodings and run a locale-poor CI image, you do not need my forty-eight hours. If your pipeline is air-gapped, a hosted extra shell is the wrong second machine, and you should keep the probe inside your own runners. If you parse binary formats, images, or truly unknown vendor encodings, UTF-8 pinning can corrupt data instead of saving it.

This is also a weak fit for notebooks that never leave one workstation. A single-locale life will keep lying to you politely, and a second environment is the entire point of the notes. Teams that already fail CI on PYTHONWARNDEFAULTENCODING have already bought the lesson I paid for slowly.

Limitations I am not going to paper over

I did not benchmark anything, and I am not claiming a spare remote shell matches production kernels, CA stores, or disk layouts. I exported LANG=C to make the ASCII failure deterministic, which is a test fixture, not a census of the internet. Python 3.15 may make UTF-8 mode the default later this year, and that will retire part of this footgun without retiring the habit of pinning encodings you actually control.

The probe does not prove a file is UTF-8; it only proves your process and your locale disagree. For vendor drops I still want a decoder that fails loudly, plus a hex dump of the first few hundred bytes. Generated code remains a sketch. If the sketch omits encoding=, I now assume it omitted something else too.

I would rather keep a ten-line probe in the repo than trust another quiet local pass. If you already have a locale-poor job, run the script there before you rewrite the parser.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.