I Blamed the CSV for 48 Hours. The Clean Box Still Spoke POSIX.

I spent two nights staring at a UnicodeDecodeError that refused to appear on my laptop. The traceback named a CSV I had already re-exported, re-saved, and hex-dumped until my eyes hurt. Have you ever trusted a local green pytest run more than a remote red log? I did, and that misplaced trust burned a full forty-eight hours.

The job itself was boring on paper: read a small UTF-8 CSV, normalize two name fields, then write JSON. It passed every test on my machine, including the run I executed immediately before I pushed. Then the same commit hit a clean Linux box and died inside open(), not inside my parsing logic, which felt oddly personal.

Hour 0–8: I interrogated the file, not the process

I treated the CSV like a crime scene, because that is what the stack frame invited me to do. file, hexdump, and a quick Python sniff all said UTF-8, with a BOM-free header and one é in a surname. Why would a decoder explode on a file that looks textbook clean?

file names.csv
# names.csv: CSV text

hexdump -C names.csv | head -n 4
# 00000000  69 64 2c 6e 61 6d 65 0a  31 2c 52 65 6e c3 a9 0a  |id,name.1,René.|

python -c "print(open('names.csv','rb').read())"

Those bytes are legal UTF-8. The c3 a9 sequence is é, not corruption, not a Windows-1252 leftover, and not a truncated character. I rewrote the exporter anyway, because blaming the producer is cheaper than blaming your own runtime. The new file still crashed on the clean box, and my laptop still passed, which should have been the clue.

Hour 8–16: I asked a model for a patch instead of a probe

I pasted the traceback into MonkeyCode's free model access and asked for a fix, not a hypothesis list. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The reply was fluent, confident, and half right, which is a dangerous combination when you are already tired.

It told me to write open(path, encoding="utf-8") everywhere, which I should have done on day one. It also suggested locale.setlocale(locale.LC_ALL, "en_US.UTF-8") at process start, which looks responsible in a blog snippet. Have you noticed how often assistants patch the symptom you pasted, rather than the environment you did not paste?

# labeled example: the incomplete "fix" I almost shipped
import locale

def configure_locale() -> None:
    # This raises locale.Error on a box that never ran locale-gen.
    locale.setlocale(locale.LC_ALL, "en_US.UTF-8")

I applied the encoding="utf-8" change to the CSV reader and felt clever for ten minutes. A later Path.read_text() call, plus a subprocess pipeline with text=True, still used the process default. The clean box kept failing, just one stack frame lower, which I misread as progress.

Hour 16–28: My laptop was lying with a hidden UTF-8 crutch

I could not reproduce locally, so I started inventing ghosts: bad Docker layers, a stale wheel, even a "corrupt" pytest cache. Does that spiral sound familiar, or is it only me? The turning point was running the same commands with a stripped environment, which finally made my laptop behave like the remote box.

# On the laptop, the lie is easy to miss.
python -c "import locale,sys; print(sys.flags.utf8_mode); print(locale.getpreferredencoding(False))"
# 1
# utf-8

env -i HOME="$HOME" PATH="$PATH" python -c "import locale,sys; print(sys.flags.utf8_mode); print(locale.getpreferredencoding(False))"
# 0
# ANSI_X3.4-1968

UTF-8 mode had been on in my interactive shell the entire time. A user-level sitecustomize.py plus PYTHONUTF8=1 in .zshrc made open() look safe. The clean server had neither crutch, so Python fell back to the POSIX locale encoding, which is ASCII in practice.

# labeled reproduction: save as repro_locale.py
from pathlib import Path

SAMPLE = Path("names.csv")
SAMPLE.write_bytes(b"id,name\n1,Ren\xc3\xa9\n")

# Fails under LANG=C when encoding is omitted.
print(SAMPLE.read_text())

env -i PATH="$PATH" LANG=C LC_ALL=C python repro_locale.py
# UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position ...

That one command did more than eight hours of file staring. If your local shell exports PYTHONUTF8, LC_ALL=en_US.UTF-8, or a conda activation script, you do not have a local reproduction. You have a comfort setting.

Hour 28–40: The free server was the first honest machine

I needed a box that did not inherit my dotfiles, so I used MonkeyCode's free server option as a scratch Linux environment. A spare VPS or a throwaway container would have taught the same lesson. The point was not the brand; the point was a process that had never seen my .zshrc.

locale
# LANG=C
# LC_ALL=C

locale -a
# C
# POSIX

python -c "import locale,sys; print(sys.version.split()[0], sys.flags.utf8_mode, locale.getpreferredencoding(False))"
# 3.12.x 0 ANSI_X3.4-1968

No en_US.UTF-8. No C.UTF-8 on that image either. locale.setlocale(..., "en_US.UTF-8") raised locale.Error, which is the failure mode the model never mentioned. Should a coding assistant be expected to know your glibc locale archive? Only if you paste locale -a, and I had not.

Locale probe I wish I had run at hour one

This is the artifact I now keep next to any text-processing job. Run it in CI and on the box that actually executes the job, not only on the laptop that authors the job.

#!/usr/bin/env bash
# labeled artifact: probe_locale.sh
set -euo pipefail

echo "=== env ==="
printf 'LANG=%s LC_ALL=%s LC_CTYPE=%s PYTHONUTF8=%s PYTHONIOENCODING=%s\n' \
  "${LANG-}" "${LC_ALL-}" "${LC_CTYPE-}" "${PYTHONUTF8-}" "${PYTHONIOENCODING-}"

echo "=== locale -a (utf/c.utf only) ==="
locale -a | grep -Ei 'utf|posix|^c$' || true

echo "=== python ==="
python - <<'PY'
import locale, os, sys
print("version", sys.version.split()[0])
print("utf8_mode", sys.flags.utf8_mode)
print("fsencoding", sys.getfilesystemencoding())
print("preferred", locale.getpreferredencoding(False))
print("stdout", getattr(sys.stdout, "encoding", None))
try:
    locale.setlocale(locale.LC_ALL, "en_US.UTF-8")
    print("setlocale en_US.UTF-8: ok")
except locale.Error as exc:
    print("setlocale en_US.UTF-8: FAIL", exc)
PY

chmod +x probe_locale.sh
./probe_locale.sh
env -i PATH="$PATH" LANG=C LC_ALL=C ./probe_locale.sh

If those two runs disagree, your test suite is not testing the runtime you ship. It is testing your shell cosmetics.

Hour 40–48: The fix was boring, the checklist was not

I stopped trying to make the server pretend it was my laptop. Text I control now declares UTF-8 at the call site, and subprocess calls do the same. Locale configuration, if I need it at all, is a documented machine image concern, not a hidden sitecustomize.py.

from pathlib import Path
import subprocess

def load_names(path: Path) -> str:
    return path.read_text(encoding="utf-8")

def run_helper(args: list[str]) -> str:
    completed = subprocess.run(
        args,
        check=True,
        text=True,
        encoding="utf-8",
        capture_output=True,
    )
    return completed.stdout

# pytest.ini — force the hostile locale in CI, not the friendly one
[pytest]
addopts = -q

# CI / Makefile target I will actually repeat
env -i PATH="$PATH" LANG=C LC_ALL=C PYTHONUTF8= pytest -q

That env -i line is the whole lesson wearing a disguise. If the suite still passes, you earned the green. If it dies on é, you found the bug before a clean box did it in production.

Decision table from the wreckage

Symptom	First check	Do not do	Do this
UnicodeDecodeError on a file you know is UTF-8	locale.getpreferredencoding(False) on that host	Re-export the CSV for the third time	Pass encoding="utf-8" at every text boundary
locale.Error: unsupported locale setting	locale -a	setlocale(..., "en_US.UTF-8") because a model said so	Use C.UTF-8 if present, or skip setlocale and declare encodings
Local green, remote red	env -i LANG=C python ...	Trust .zshrc, direnv, or conda	Add a hostile-locale CI job
text=True subprocess blows up	child stdout encoding	Set PYTHONIOENCODING only on your laptop	Pass encoding="utf-8" into subprocess.run
Assistants keep rewriting the parser	Did you paste locale and sys.flags.utf8_mode?	Ask for a patch first	Ask for a probe script first

What broke, and what I would repeat

The CSV was never dirty. My laptop locale was the liar, and the model amplified that lie by patching call sites I showed it. The clean server was honest because it had no UTF-8 locale and no UTF-8 mode. Would I still use an assistant on this class of bug? Yes, but I would paste probe output before I accept a patch.

What I would repeat:

1. Run probe_locale.sh on the execution host before I read another traceback.
2. Ask the model for a reproduction command under LANG=C, not for a rewrite of my parser.
3. Keep one CI job that starts from env -i so dotfiles cannot hide encoding bugs.
4. Treat encoding="utf-8" as part of the function contract, not as a style flourish.

What I would not repeat:

• Shipping locale.setlocale to a machine that never ran locale-gen.
• Believing a laptop that exports PYTHONUTF8=1.
• Editing the data file because the stack frame named the data file.

Limitations, and who should skip this approach

This workflow assumes you control the text format and can pin UTF-8 at the boundary. If you consume unknown legacy encodings, you still need an explicit detection policy, and encoding="utf-8" will be the wrong hammer. Hostile-locale tests also will not catch filesystem encoding bugs on Windows, where the rules are different and LANG=C is not the story.

Skip the free-model-plus-scratch-server loop when the corpus cannot leave your network, or when the job needs a guaranteed image identical to production. A free server is a useful liar-detector for locale and dotfiles. It is not a substitute for the AMI, base image, or distroless digest you actually ship.

The assistant side has limits too: it cannot see sitecustomize.py unless you show it, and it will happily invent a locale name that glibc does not have. I still like having free model access in the same loop as a clean box, because the box falsifies the model. If you want that pairing without standing up your own VM, I used MonkeyCode's free server option as the honest environment in the second half of this hunt; any scratch Linux host would have worked.

Forty-eight hours later, the CSV is unchanged, the parser is smaller, and the only new test is LANG=C. That is a painfully cheap lesson. Why did I wait until the clean box said it out loud?