The script was eight years old.
It read the supplier's price file
and wrote a report.
with open("prices.csv") as f:
No encoding.
Nobody had ever needed one.
On Linux
it read the file as UTF-8,
because that was the locale.
On the Windows box
that ran it every night,
it read the file as cp1252,
because that was the locale there.
The supplier sent cp1252,
pound signs and all,
and on Windows
it happened to work.
That is the thing about
a default nobody chose.
It works
until the machine changes
or the default does.
The default is changing.
PEP 686
makes Python's UTF-8 mode
the default,
starting with Python 3.15.
Text files opened
without an encoding
will be read and written as UTF-8
on every platform,
whatever the locale says.
On Linux and macOS
most people will notice nothing.
On Windows
that line changes meaning.
A pound sign in cp1252
is the single byte A3.
In UTF-8
that byte cannot start a character.
So the nightly script
stops on its first pound sign
with a UnicodeDecodeError.
The other direction is quieter.
A report written by the new Python
and opened by an old tool
that still expects cp1252
shows £
where the pound sign was.
You can turn the new mode off.
PYTHONUTF8=0,
or -X utf8=0.
That buys time,
not a fix.
The fix is to stop relying
on the default at all.
Python has helped with this
since 3.10.
Run your tests with
-X warn_default_encoding
and every open
without an encoding
raises an EncodingWarning
with the line number.
Then go through them.
Files you control,
encoding="utf-8".
Files that really are
in the local Windows encoding,
say so.
encoding="locale"
keeps the old behaviour,
on purpose this time.
Files from suppliers and partners,
write down what they send,
and open them that way.
Do it before the upgrade,
while the old version
still runs the jobs,
so every change you make
can be tested
against the behaviour you know.
Then upgrading
changes nothing at all.
Which is the point.
A default is somebody else's guess
about your data.
Python has changed its guess.
Make your own,
and write it down.
– Serguey Asael Shinder
Top comments (0)