My Python script read a file fine on Linux and crashed on Windows. The bug was open() without encoding

The same script worked on my Linux server and raised UnicodeDecodeError on a Windows laptop. Same file, same code, same Python version. Here is why, and the one-argument fix.

Reproduce it

Create a UTF-8 file with a few non-ASCII characters:

from pathlib import Path

Path("notes.txt").write_bytes("café ✓ naïve".encode("utf-8"))

Now read it the lazy way:

text = open("notes.txt").read()
print(text)

On most Linux and macOS machines this prints café ✓ naïve. On a Windows machine with a Western European locale you get one of two things:

• Mojibake: café ... because the bytes were decoded with the wrong table.
• A crash: UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f ... when the file contains a byte that the Windows code page does not define.

Nothing in your code changed. The environment did.

Why it happens

When you call open() without encoding=, Python uses the locale's preferred encoding. You can see what it is on your machine:

import locale
print(locale.getpreferredencoding(False))

On a typical modern Linux box that prints UTF-8. On a Windows machine it is often cp1252 (or another legacy code page, depending on the region). So the exact same line of code means "read as UTF-8" on one machine and "read as cp1252" on another. The Python docs say this directly in the description of the encoding parameter of open(): the default is platform dependent.

The fix

Always say what encoding you mean:

with open("notes.txt", encoding="utf-8") as f:
    text = f.read()

with open("out.txt", "w", encoding="utf-8") as f:
    f.write(text)

The same goes for pathlib, which has the same trap:

from pathlib import Path

text = Path("notes.txt").read_text(encoding="utf-8")
Path("out.txt").write_text(text, encoding="utf-8")

Same for csv, json.load(open(...)), and anything else that opens a text file for you: pass encoding="utf-8" to the open() underneath it. (Binary mode, "rb" / "wb", has no encoding, so it is not affected.)

Catch the missing ones automatically

You do not have to grep your whole project by hand. Python 3.10 added an EncodingWarning that fires every time a text file is opened without an explicit encoding. It is off by default; turn it on with a flag:

python -X warn_default_encoding your_script.py

Every open() call that relies on the default now prints a warning with the file and line number. This was added by PEP 597, which is worth a read if you want the full reasoning.

A stopgap, not a fix

If you cannot edit the code (a third-party script, for example), you can force UTF-8 mode for the whole process:

python -X utf8 your_script.py
# or set it once in the environment:
set PYTHONUTF8=1        # Windows cmd
export PYTHONUTF8=1     # Linux / macOS

This makes the default encoding UTF-8 regardless of the locale. It is handy, but it is a setting on the machine, not a property of your code, so the next person who runs your script without it hits the same bug. PEP 686 proposes making UTF-8 mode the default in a future Python version (check the PEP for the exact release), but until you control every interpreter your code runs on, being explicit is the safe habit.

Checklist

1. Search your code for open(, read_text(, write_text( and look for a missing encoding=.
2. Add encoding="utf-8" to each text-mode call.
3. Run your tests once with python -X warn_default_encoding to find the ones you missed.
4. Keep PYTHONUTF8=1 only as a safety net for code you do not own.

What is the weirdest "works on my machine" bug an OS default has caused for you?