Why your Python tool corrupts text on Windows, and why chcp 65001 doesn't fix it

I lost an afternoon to a config file last week. The graph name was right there in the JSON, spelled exactly the way the CLI wanted it, and the CLI kept saying the graph didn't exist.

Turns out the file wasn't being read wrong...... it was being decoded wrong. The name that came out the other end had a couple of extra characters in it, and nothing raised an error to tell me.

Then the same class of bug turned up in an updater. It printed "Up to date" while two reader threads died with charmap codec can't decode byte 0x81. The check thought everything was fine, which is the worst part of it.

Both are one thing: Python's default text encoding on Windows is not UTF-8. Not for files, and not for subprocess pipes.

What Python actually uses by default

open("config.json") with no encoding= argument decodes with whatever locale.getpreferredencoding(False) returns. On Windows that's the ANSI code page. cp1252 on most en-US machines, cp936/GBK on zh-CN, cp932 on ja-JP, cp1251 on ru-RU.

Pipes are the same story. subprocess.run(cmd, text=True) with no encoding= hands the pipe to TextIOWrapper, which falls back to that same locale codec. So the parent process decodes the child's UTF-8 output with cp1252 and hopes for the best.

If your files are UTF-8 (most config, most code, most modern tooling), you have a codec mismatch. What happens next depends on the bytes, and there are four flavors.

The four ways it breaks

1. Mojibake, no exception

I encoded café— to UTF-8, which gives 63 61 66 c3 a9 e2 80 94, then decoded those bytes as cp1252 and got café—. cp1252 maps almost every byte to something, so nothing raises. Your dict key is now café and the lookup for café misses. That was my graph-not-found bug, and silent corruption is worse than a crash because you get no pointer to it.

2. A hard UnicodeDecodeError

Same bytes, decoded as GBK: 'gbk' codec can't decode byte 0x94 in position 7: incomplete multibyte sequence. GBK wants a valid lead byte followed by a valid trail byte, and a UTF-8 sequence is neither.

3. The single-byte crash

cp1252 defines no character for 0x81, 0x8D, 0x8F, 0x90, 0x9D. One of those lands in the stream and you get 'charmap' codec can't decode byte 0x81 in position 0: character maps to <undefined>. Small detail that explains why this shows up in normal files: 0x81 is a UTF-8 continuation byte, so it appears in the middle of ordinary characters (U+2041 is E2 81 81). You don't need exotic text to hit it.

4. A text file gets called "binary"

Plenty of tools sniff the first N bytes to guess whether a file is binary, and 1000 is a popular sample size. Cut a UTF-8 file at byte 1000 and you can split a multibyte character in half: 'utf-8' codec can't decode byte 0xe6 in position 999: unexpected end of data. The rest of the file decodes cleanly and there is no NUL byte anywhere, but the tool decides it's binary and refuses to display it. I've had that happen on a plain CJK text file.

chcp 65001 won't save you

I tried it. Console code page was 437, I ran chcp 65001, and locale.getencoding() still returned cp1252. getpreferredencoding(False) still returned cp1252. The Popen(text=True) reader still handed me café.

chcp sets the console code page. Python's default text encoding comes from the system ANSI code page (GetACP), which is a different thing. And for pipes the console isn't involved at all...... there's no console anywhere in that path.

Fix it in the source

Pin the encoding at every point where bytes become text. That's the whole fix, and it's boring on purpose:

import json, subprocess, sys
from pathlib import Path

# text files: say what they are
with open("config.json", encoding="utf-8") as f:
    data = json.load(f)

text = Path("README.md").read_text(encoding="utf-8")
Path("out.md").write_text(text, encoding="utf-8")

# child processes: text mode, an explicit encoding, and a forgiving handler
p = subprocess.run(["git", "log", "--oneline"], capture_output=True,
                   text=True, encoding="utf-8", errors="replace")
print(p.stdout)

# if it's your own stdout that's misbehaving
sys.stdout.reconfigure(encoding="utf-8", errors="replace")

A couple of details that bite later:

• Use encoding="locale" when the file really is in the platform encoding (3.10+). Don't guess.
• errors="replace" never raises, which is also why it can hide a real mismatch. Reach for errors="surrogateescape" if you need to round-trip arbitrary bytes without losing any.

To find the sites in code you already have, run with PYTHONWARNDEFAULTENCODING=1 (or python -X warn_default_encoding). Python 3.10+ emits an EncodingWarning everywhere the default is being relied on. Run your tests once with it enabled, fix the hits, done. That's the PEP 597 answer to "how do I find these", and it beats grepping for open(.

If you can't patch the tool

Third-party CLI, vendor binary, something you don't own: set PYTHONUTF8=1 in that process's environment. That's UTF-8 mode (PEP 540, available since 3.7). It makes open() default to UTF-8, switches the filesystem encoding to UTF-8, and, the part that matters here, makes locale.getpreferredencoding() return utf-8 so pipe decoding gets the right codec.

Same script, both settings, on Windows 11 with Python 3.11:

PYTHONUTF8=0  getpreferredencoding: cp1252  |  Popen(text=True): 'café—'
PYTHONUTF8=1  getpreferredencoding: utf-8   |  Popen(text=True): 'café—'

Notice locale.getencoding() stays cp1252 in both rows. UTF-8 mode doesn't change the machine's ANSI code page, it changes what Python asks for.

Two honest caveats, because this is where people get hurt:

• UTF-8 mode is not a codec fix. If the child process actually emits GBK bytes, you still fail. You just fail with utf-8 named in the traceback instead of cp1252. errors="replace" is the crash-proof option. UTF-8 mode covers the common case, it doesn't make you right.
• I wouldn't set it machine-wide. It changes the default for every Python process on the box and flips os.fsencode/os.fsdecode to UTF-8, so anything that legitimately exchanges ANSI or OEM encoded data with a Windows component can start producing mojibake. Per process, per venv, or per CI job is the sane scope. Also, PYTHONIOENCODING does not cover pipes: I had it set to utf-8 and the pipe still decoded as cp1252.

For what it's worth, this is going away. PEP 686 makes UTF-8 the default in 3.15, and the same PEP warns that the change can surface UnicodeError and mojibake in code that was leaning on the old behavior without saying so. Expect a wave of these reports when it lands.

The binary-sniff false positive

Don't decide "binary" from a truncated sample. Check for a NUL byte (that's what git does in its 8000-byte scan) and let an incremental decoder deal with the edge, so you never split a character:

import codecs

def looks_binary(path, sniff=8000):
    with open(path, "rb") as f:
        head = f.read(sniff)
    if b"\x00" in head:
        return True
    dec = codecs.getincrementaldecoder("utf-8")()
    try:
        dec.decode(head)   # a partial char at the edge simply isn't flushed
        return False
    except UnicodeDecodeError:
        return True

That returns False for UTF-8 text no matter where the boundary lands, CJK included.

Wrapping up

I'm fairly sure this is the whole class of the bug, but I've only hit it on Windows, so your mileage may vary on a Linux box with a non-UTF-8 locale (same mechanism, much rarer default). If a tool prints a name that looks correct and still can't find it, hexdump the config and hexdump the value it's comparing against. The extra bytes will be sitting right there. Hope this saves somebody the afternoon it cost me.

Where I ran into it: langchain-ai/langgraph#8665 (config reads on a CJK locale), NousResearch/hermes-agent#97322 (the cp1252 pipe reader thread), NousResearch/hermes-agent#86187 (valid UTF-8 flagged as binary).