mattleeee

Posted on Oct 13 Originally published at hkcode.dpdns.org

UTF-8 vs GBK: The Root Cause of Cross-Machine Script Failures

#utf8 #gbk #codepage #pythonioencoding

A script that runs fine on your laptop and dies on the server is almost never a logic bug. It is an encoding bug that was always there, dormant, waiting for a machine with a different codepage. Windows makes this especially easy to trip over because the default encoding is not a constant — it is a property of the OS locale, and it leaks into every open() call, every print(), and every subprocess you spawn.

Here is the failure mode, the mechanism, and the fixes that actually hold up in production.

The Two Machines

The scenario is common enough to be a template:

Dev box: Windows 11, "Beta: Use Unicode UTF-8 for worldwide language support" enabled. sys.stdout.encoding reports utf-8, locale.getpreferredencoding() reports utf-8.
Production: Chinese Windows Server 2016, default locale. Codepage 936 (GBK). sys.stdout.encoding reports cp936, locale.getpreferredencoding() reports cp936.

Same Python version, same code. The dev box is in a UTF-8 bubble; the server is not. Everything that depends on the ambient encoding flips behavior between them.

What `open()` actually does

When you call open(path, "w") without an encoding argument, Python uses locale.getpreferredencoding(False). On the dev box that is utf-8. On the server that is cp936. This single default is the source of most cross-machine breakage.

# This is not portable. The encoding is decided by the machine.
with open("report.txt", "w") as f:
    f.write("季度汇总\n")

On the dev box, report.txt is UTF-8. On the server, it is GBK. If a downstream tool on either side assumes the other encoding, you get mojibake or a decode error. The file is byte-for-byte different depending on where it ran, and nothing in the code says so.

What `print()` actually does

print() writes to sys.stdout, which is a TextIOWrapper whose encoding is derived from the console codepage. On the server, that is cp936. If you print a character that GBK cannot represent, you get:

UnicodeEncodeError: 'gbk' codec can't encode character '\u2713' in position 0: illegal multibyte sequence

A checkmark, an emoji, a CJK extension character, a box-drawing glyph — any of these will crash a print() on the server and do nothing on the dev box. This is the failure that surfaces first because it is loud.

The Failure Cases

Case 1: `print()` crashes on the server

print("Done ✓")  # fine on UTF-8 console, UnicodeEncodeError on cp936

The traceback points at print, which makes it look like a stdout problem. It is. The console cannot represent ✓ in GBK, and Python raises rather than substituting.

Case 2: File paths get mangled

This one is quieter and worse. Windows paths are UTF-16 internally, but when they cross a codepage boundary — through a .bat file, an environment variable, or a subprocess argument — they get encoded and decoded using the active codepage. A path containing a character outside GBK, or a path that was written as UTF-8 in a bat file and read as GBK, comes back as garbage:

# Path read from a config file that was saved as UTF-8
# but the interpreter decoded it as cp936
path = "C:\\数据\\报表.csv"   # becomes "C:\\鏁版嵁\\鎶ヨ〃.csv"

The file exists. The path string does not match it. FileNotFoundError on a file you can see in Explorer.

Case 3: Bat files corrupt non-ASCII

A .bat file with Chinese characters in it is interpreted using the console codepage at the time cmd.exe reads it. If the file is saved as UTF-8 (with or without BOM) and the console is cp936, cmd.exe misreads every multibyte character. echo 开始 prints 寮€濮�, and worse, any command built from those characters operates on the wrong path or argument.

Case 4: Subprocess output decoded with the wrong codec

import subprocess
out = subprocess.run(["some_tool.exe"], capture_output=True).stdout
text = out.decode()  # uses locale.getpreferredencoding() -> cp936 on server

If some_tool.exe emits UTF-8, this decode produces mojibake or raises. If it emits GBK and you decode as UTF-8, same result in reverse. The subprocess inherits no encoding contract, so you have to impose one.

The Survival Rules

These five rules eliminate the entire class of bug. They are not stylistic preferences; each one closes a specific hole.

Rule 1: Always pass `encoding=` to `open()`

Never rely on the locale default. Be explicit at every call site.

import io

# Reading
with open("input.csv", "r", encoding="utf-8", newline="") as f:
    rows = f.read().splitlines()

# Writing
with open("output.csv", "w", encoding="utf-8", newline="") as f:
    f.write("name,value\n")
    f.write("季度,42\n")

For CSV specifically, pass the encoding through to the csv module's file handle, not to the reader/writer:

import csv

with open("data.csv", "w", encoding="utf-8", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["symbol", "note"])
    writer.writerow(["AAA", "正常"])

newline="" matters on Windows: without it, the csv module writes \r\r\n because the text layer translates \n and the writer already emits \r\n.

If you need to interoperate with Excel on a Chinese Windows box, utf-8-sig (UTF-8 with BOM) is the pragmatic choice for CSV — Excel will open it correctly without an import wizard:

with open("for_excel.csv", "w", encoding="utf-8-sig", newline="") as f:
    ...

Be aware this is a CSV/Excel concession. For logs and internal data, plain UTF-8 is correct.

Rule 2: Set `PYTHONIOENCODING=utf-8` in launchers

Setting this environment variable forces sys.stdin, sys.stdout, and sys.stderr to UTF-8 regardless of the console codepage. In a bat launcher:

@echo off
set PYTHONIOENCODING=utf-8
set PYTHONUTF8=1
python main.py

PYTHONUTF8=1 enables UTF-8 mode, which additionally forces locale.getpreferredencoding() to return utf-8, making even the un-annotated open() calls behave consistently. Combining both is belt and suspenders, and it is cheap.

There is a catch: when stdout is redirected to a file or a pipe, the console codepage no longer matters and the encoding is whatever PYTHONIOENCODING says. When stdout is an interactive console, the console still has to render the characters. Setting PYTHONIOENCODING=utf-8 stops the UnicodeEncodeError, but the console may show boxes for characters it cannot display. That is a cosmetic limitation of the console, not a data corruption — the bytes written to a redirected file are correct UTF-8.

You can verify what the interpreter sees:

import sys, locale
print("stdout:", sys.stdout.encoding)
print("preferred:", locale.getpreferredencoding(False))
print("utf8 mode:", sys.flags.utf8_mode)

Run this on both machines before you debug anything else. It tells you immediately which regime you are in.

Rule 3: Wrap console prints in a GBK-safe fallback

Even with PYTHONIOENCODING set, you cannot always control how your script is launched. A scheduled task, a service wrapper, or a colleague's cmd window may not set the variable. A defensive print wrapper costs little and never crashes:

import sys

def safe_print(*args, **kwargs):
    """Print without ever raising UnicodeEncodeError.

    Falls back to a lossy ASCII representation when the target
    stream cannot encode the text.
    """
    stream = kwargs.get("file", sys.stdout)
    encoding = getattr(stream, "encoding", None) or "ascii"
    text = " ".join(str(a) for a in args)
    try:
        text.encode(encoding)
    except (UnicodeEncodeError, LookupError):
        text = text.encode(encoding, errors="replace").decode(encoding)
    print(text, **kwargs)

safe_print("Done ✓") on a cp936 console prints Done ? instead of raising. The information about the checkmark is lost, but the script continues. For status output that is the right trade. For data you intend to parse later, do not print it — log it (Rule 4).

If you want to preserve more information, you can map known glyphs to ASCII equivalents before falling back:

_FALLBACKS = {"✓": "[OK]", "✗": "[FAIL]", "→": "->", "—": "-"}

def safe_print(*args, **kwargs):
    stream = kwargs.get("file", sys.stdout)
    encoding = getattr(stream, "encoding", None) or "ascii"
    text = " ".join(str(a) for a in args)
    for ch, repl in _FALLBACKS.items():
        if ch in text:
            try:
                text.encode(encoding)
            except UnicodeEncodeError:
                text = text.replace(ch, repl)
    try:
        text.encode(encoding)
    except (UnicodeEncodeError, LookupError):
        text = text.encode(encoding, errors="backslashreplace").decode(encoding)
    print(text, **kwargs)

Rule 4: Write log files as UTF-8 regardless of console codepage

Logs are data. They should not depend on the console. Configure logging with an explicit UTF-8 file handler and, if you also log to the console, let the console handler degrade gracefully.

import logging
import sys

logger = logging.getLogger("app")
logger.setLevel(logging.INFO)

# File handler: always UTF-8, independent of the console.
fh = logging.FileHandler("app.log", encoding="utf-8")
fh.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(message)s"))
logger.addHandler(fh)

# Console handler: keep it, but tolerate an unencodable console.
ch = logging.StreamHandler(sys.stdout)
ch.setFormatter(logging.Formatter("%(levelname)s %(message)s"))
logger.addHandler(ch)

logger.info("处理完成，共 %d 条记录", 128)

The file always gets clean UTF-8 bytes. The console handler may still raise on a cp936 console if the message contains an unencodable character — which is why Rule 2 (set PYTHONIOENCODING) and Rule 3 (safe wrapper) matter. If you want the logging module itself to never raise, wrap the stream:

import io

class SafeStream(io.TextIOBase):
    def __init__(self, raw):
        self._raw = raw
        self.encoding = getattr(raw, "encoding", None) or "ascii"

    def write(self, s):
        try:
            s.encode(self.encoding)
        except (UnicodeEncodeError, LookupError):
            s = s.encode(self.encoding, errors="replace").decode(self.encoding)
        return self._raw.write(s)

    def flush(self):
        self._raw.flush()

ch = logging.StreamHandler(SafeStream(sys.stdout))

This is the same idea as safe_print, applied at the stream level so every handler benefits.

Rule 5: Keep bat files ASCII-only

This is the rule people resist, and it is the one that saves the most pain. A .bat file is read by cmd.exe under the active console codepage. There is no reliable way to make a bat file containing non-ASCII text behave identically across codepages — the chcp command changes the console codepage but does not retroactively fix how cmd.exe parsed the bytes it already read from the file.

Keep every .bat file pure ASCII. Move any non-ASCII string into a Python script or a UTF-8 data file that Python reads with an explicit encoding.

@echo off
REM ASCII only. No Chinese characters in this file.
set PYTHONIOENCODING=utf-8
set PYTHONUTF8=1
python "%~dp0\run_task.py" %*

If you must reference a path with non-ASCII characters, do it from Python, not from the bat file:

# run_task.py
from pathlib import Path

base = Path(r"C:\data")          # ASCII in the source, safe
target = base / "季度报表.csv"    # non-ASCII lives in Python source,
                                  # which is UTF-8 by default (PEP 3120)
with open(target, "r", encoding="utf-8", newline="") as f:
    ...

Python source files are UTF-8 by default since Python 3, so non-ASCII literals in .py files are safe. The bat file is the only place where the bytes are re-interpreted by a non-Python parser.

A Checklist

Before shipping a script that will run on a machine you do not control:

Every open() has an explicit encoding=.
Every subprocess capture is decoded with an explicit encoding, or you use text=True, encoding="utf-8".
The launcher sets PYTHONIOENCODING=utf-8 and PYTHONUTF8=1.
Console output goes through a fallback that cannot raise.
Log files are UTF-8, always.
Bat files contain only ASCII.
You ran the sys.stdout.encoding / locale.getpreferredencoding probe on both machines.

The pattern behind all of these is the same: stop letting the ambient locale decide anything. Encoding is a property of your program's data contract, not of the machine it happens to run on. Once you make it explicit, the dev box and the server stop disagreeing.

More notes like this ship every week on this site.

Top comments (1)

Emma Wingding • Oct 13

Really solid write-up — this one has bitten me more times than I'd like to admit.

Two small additions from the trenches:

Your Rule 3 fallback with "?" works, but it throws away information. A middle ground is to encode with errors="backslashreplace", so the checkmark shows up in the log as \u2713 instead of vanishing entirely. The log stays greppable, and you can tell a glyph was there rather than plain text.
Even once the codec is sorted, Unicode normalisation can still bite across machines: the same logical string can arrive as NFC on one box and NFD on another, so equality checks and dedup silently disagree. Running input strings through unicodedata.normalize("NFC", ...) once at ingestion tends to close that hole.

Excellent checklist though — the "probe both machines first" step alone would have saved me a memorable afternoon.