48-Hour Field Notes: I Printed the Preferred Encoding. That Was Not the Open Default.

I spent the first evening sure the patch was finished, because the sample opened and the assertion printed the name I expected. Have you ever watched a text-mode read succeed on your laptop and then die the moment a clean job starts? I had a short traceback, a green local run, and a bad guess that the suggested edit had covered the real open. The bytes were valid UTF-8, so why did the clean job talk about an ASCII codec at all?

What I tried before I blamed the file

I started where most of us start, with the diff, the local test command, and a second read of the traceback. The suggested comment said the file was already UTF-8, so the decode could stay implicit and nobody would mind. I reran the same pytest target in the editor terminal I had used all afternoon, and that run passed again. Would you have stopped there too, if the only red line lived on a job you could not open yet?

Here is the sequence I burned time on, before a second encoding print changed what I thought the suggestion had proved:

1. I checked the sample bytes and confirmed they were UTF-8, with no BOM and no stray Latin-1 byte sitting in the name.
2. I ran that pytest target in the editor terminal, then again in a fresh local shell, and both of those runs stayed green.
3. I copied the job command onto my laptop without the job environment, which hid the real difference for another hour.
4. I asked for another rewrite, pasted only the failing assertion, and received a new expected string that never mentioned locale.

That fourth step felt like progress while I was typing, and it did not move the actual failure one inch. A model can only repair the world you actually paste into the prompt, and I had pasted the wrong world. I had a passing local command beside a failing remote line, but I never pasted the locale or the interpreter flags. The suggestion kept rewriting the assertion, while the open underneath that assertion stayed implicit and unpinned.

What broke once the shell got smaller

The break got interesting when I stopped borrowing my laptop environment and used a shell closer to a fresh image. I exported LANG and LC_ALL as C, then printed every encoding helper the standard library was willing to show me. Python text mode does not mean UTF-8 on every machine, and I wish I had written that sentence on the ticket first. In the C locale, the encoding an implicit open consults is often ASCII-compatible, so a non-ASCII name raises instead of returning a string.

A follow-up suggestion told me to trust the preferred encoding, because UTF-8 mode would make every text open safe. Have you ever shipped that advice after one print succeeded, without asking which helper open actually consults? I had started down that path, and it is the miss I would not repeat on the next clean job. PEP 540 describes UTF-8 mode, but I will not pretend one flag repairs Path.read_text on every release you might boot.

Two helpers that do not answer the same question

locale.getpreferredencoding(False) is the print I had trusted, and it can follow UTF-8 mode rather than the raw locale. locale.getencoding(), where your interpreter has it, is documented as the locale encoding with that override ignored. Recent open() docs point at that locale encoding when you omit the argument, which is a different fact from a cheerful print. That split is why a preferred-encoding line can approve a patch the clean job still rejects.

I am not going to freeze a version number into this note as if the default could never move again. Re-read the open() and locale pages for that job's interpreter before you quote a codec name. If those two helpers disagree, neither one gets to approve the merge on its own. If a build enables UTF-8 mode by default, a LANG=C export may still leave the flag on, and the print is the fact that matters.

The repro I would keep in the repo

This file is a procedure for you to run, not a private job log and not a benchmark I want quoted. It writes the UTF-8 bytes for a short non-ASCII name without asking your shell to pass those bytes through. It prints the flag, both helpers when one of them exists, and the implicit read beside an explicit read. Save it as repro_locale_open.py, run the two commands, and paste both transcripts into the ticket before another rewrite.

import locale
import sys
from pathlib import Path

def encoding_open_would_consult():
    getencoding = getattr(locale, "getencoding", None)
    if getencoding is not None:
        return getencoding()
    return locale.getpreferredencoding(False)

sample = Path("sample_name.txt")
sample.write_bytes(bytes([74, 111, 115, 195, 169, 10]))

print("utf8_mode:", sys.flags.utf8_mode)
print("preferred:", locale.getpreferredencoding(False))
print("open_consults:", encoding_open_would_consult())

try:
    print("implicit:", sample.read_text())
except UnicodeDecodeError as exc:
    print("implicit failed:", type(exc).__name__)

print("explicit:", sample.read_text(encoding="utf-8"))

env -u PYTHONUTF8 LANG=C LC_ALL=C python3 repro_locale_open.py
env PYTHONUTF8=1 LANG=C LC_ALL=C python3 repro_locale_open.py

The first command is the clean-job shape I care about when UTF-8 mode is actually off. The second command shows whether the flag changed, and it is not a declaration that the implicit read is fixed. The explicit read is the path I would repeat when the file contract really is UTF-8 and not a mix of legacy bytes. I would not install a locale package just to obtain a prettier name than C for this check.

The subprocess cousin of the same default

I almost missed the twin, because the file open was loud and a captured tool can fail one frame lower in the stack. subprocess.run with text=True and no encoding decodes the pipe through that same process default. A model that only sees your assertion may rewrite the expected text while the decode sits underneath, untouched. Have you reviewed a green suggestion that never touched the call which actually reads the bytes?

import subprocess

proc = subprocess.run(
    [
        "python3",
        "-c",
        "import sys; sys.stdout.buffer.write(bytes([74, 111, 115, 195, 169, 10]))",
    ],
    check=True,
    capture_output=True,
    text=True,
    encoding="utf-8",
)
print(repr(proc.stdout))

I would repeat that shape when the child contract is raw UTF-8 bytes and the parent process owns the decode. I would not pass errors="replace" just to paint the job green, because replacement hides the byte that says the contract is wrong. If you need a lossy display path for humans, keep it out of the test that claims to own the file format. A child print of a non-ASCII string is a different failure, since that child may die while encoding stdout before your parent decode even starts.

The checks I would repeat next time

I do not want another evening where "it passed here" is the whole review comment I leave on the diff. Before I accept a suggested text-mode patch, I want four checks that fit in a note, not a platform migration. Would I skip any of them if the deadline were tonight and the sample looked boring? I would still keep the first two, because those are the ones that separate a real pin from a lucky default.

• Print sys.flags.utf8_mode, locale.getpreferredencoding(False), and locale.getencoding() when it exists, inside the process that opens the file.
• Run once with PYTHONUTF8 unset and LANG=C, then again with PYTHONUTF8=1, and save both transcripts beside the diff.
• Pass encoding="utf-8" on every read_text, open, and subprocess text call that claims the file contract is UTF-8.
• Keep this one-file sample in the repo so the next clean job can fail the same way without borrowing the editor locale.

Check	What I compare	What good means here	What I will not infer
UTF-8 mode flag	Laptop process vs clean job	Both transcripts include the flag	That a green run pinned the encoding
Preferred vs locale encoding	Same process, both helpers	Any disagreement is written down	That the preferred print is what open uses
Implicit read	LANG=C, with UTF-8 mode off if you can get it	A non-ASCII sample may raise	That the sample file itself is corrupt
Explicit read	Same bytes, encoding="utf-8"	Same string with the flag on or off	That every external file is UTF-8

Where a free model pass and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I did not need a new model family to see this class of bug, and I will not pretend a product measured it. What I needed was a frozen prompt replay against the captured transcripts, plus a machine that did not inherit my editor locale. MonkeyCode's free model access and free server option matter in that narrow replay, as a clean room rather than a magic fix.

I am treating both as operator-stated availability, not as a quota, a hardware list, or a permanent offer you can plan a quarter on. I could not verify a token allotment, a model list, or a time limit from anything I would cite today, so I will not invent them. The workflow I would actually repeat is boring, which is why I trust it more than another assertion that only matches my laptop.

1. Save the failing command, both encoding prints, and the traceback into one text file before you ask anyone for a rewrite.
2. Ask which calls inherit the process default, and require a diff that passes encoding on the call that owns the file.
3. Apply that patch on a clean server, not in the shell where the editor has been exporting a UTF-8 locale all week.
4. Re-run the two env commands above, and reject the patch if you only proved it under the luckier flag.

A free server is useful here because it is a clean room, not because it sits closer to your production fleet than a container. A free model pass is useful because I can iterate on one frozen prompt while I still doubt the answer it just gave me. If either option is unavailable when you read this, those same two commands in any Linux container still teach the lesson. You should not block the encoding fix on a signup wall, and you should read the current MonkeyCode docs for what is actually included before you depend on either free option.

What this note does not prove

This note does not prove that every clean image uses locale C, and it does not prove that UTF-8 is the right contract for every file. Some logs are Latin-1, some streams are raw bytes on purpose, and some libraries ignore the encoding argument you were sure you had passed. I also did not measure runtime, token cost, or how often a model mentions locale unprompted, so please do not quote this page as a benchmark.

You should not use this replay if the sample contains private data, because a debugging convenience is not a data-processing agreement you can assume. You should also skip a model-only review when the patch touches authentication, filesystem paths, or any branch you cannot rerun from a fixture. I would not add errors="ignore" to silence locale C, since a green job that drops bytes is a different bug wearing a passing badge. This procedure is for a POSIX job you can start with env, not a claim about Windows code pages or a browser upload form.

What I would repeat tomorrow

I would keep the one-file repro, both helper prints, and an explicit encoding on every text open that claims a UTF-8 contract. I would paste the clean-job flag and both encoding lines into the prompt before I ask why a laptop run went red somewhere else. I would treat "the preferred encoding looked fine" as a hypothesis, not a merge comment, until an implicit read has been tried with the smaller locale. I would still want that explicit argument after the trial passes, because the next image can change the default without changing your diff.