UTF-8 vs. cp932: Why Some Bugs Only Show Up on Japanese Windows

"It works fine on my Mac, but it crashes the moment we hand it to Windows" is a report a lot of developers eventually hear. More often than not, the culprit is a character encoding mismatch. This post walks through the basics of text encoding, then looks at a real build-pipeline crash this project ran into, and the test written to make sure it never happens again.

What an Encoding Actually Is

Note: a character encoding is the lookup table a computer uses to convert characters into bytes for storage or transmission. The same character can turn into a completely different byte sequence depending on which encoding is used.

When a person sees the letter "A" or a Japanese character like "あ", the computer underneath is storing that as some sequence of bytes. Which byte sequence gets produced depends entirely on the encoding in use.

"あ".encode("utf-8")   # b'\xe3\x81\x82' (3 bytes)
"あ".encode("cp932")   # b'\x82\xa0' (2 bytes)

The same character produces different bytes under UTF-8 versus cp932 (the encoding used by the Japanese locale on Windows — an extension of Shift_JIS). When the encoding used to write a string and the encoding used to read it back don't match, you get garbled text or an outright error.

UTF-8 can represent almost every character in existence, emoji included. cp932 covers Japanese text — kanji, kana, full-width punctuation — perfectly well, but it does not cover most emoji. That gap is exactly where this bug lived.

"あ".encode("cp932")   # succeeds: b'\x82\xa0'
"✅".encode("cp932")   # UnicodeEncodeError

Japanese text encodes into cp932 without any trouble. Emoji don't — cp932's lookup table simply has no entry for them, so the conversion raises UnicodeEncodeError the instant it's attempted. The precise framing matters: it isn't "Japanese text breaks things," it's "characters outside cp932's table break things," and emoji happen to be the most common offender.

Why It Only Breaks on Windows

When Python captures a subprocess's output through a pipe without an encoding explicitly specified, it falls back to the platform's default encoding, which is locale-dependent. On macOS and Linux, that default is essentially always UTF-8, so a string containing emoji sails through without complaint. On Japanese-locale Windows, the default is cp932, so the exact same code crashes with UnicodeEncodeError the moment it tries to write that same string.

The nasty part is that this never reproduces on a Mac development machine. The bug only becomes visible once the code actually runs on a Windows box.

The Incident: A Crashed Pre-Build Gate

This app's build script, build_app.py, runs a pre-build gate that shells out to tools/bump_version.py --check as a subprocess to verify version-number consistency:

r = subprocess.run(
    [sys.executable, "tools/bump_version.py", "--check"],
    cwd=ROOT, capture_output=True,
)

bump_version.py printed a success message containing an emoji (✅ and similar) to stdout. On macOS, running this gate never surfaced a problem. On Japanese Windows, though, capture_output=True meant the child process's stdout got converted to cp932, and the emoji triggered UnicodeEncodeError, crashing the subprocess. build_app.py then misreported this as a failed version-consistency check — a confusing error message that pointed at the wrong problem and cost real time to trace back to its actual cause.

A Two-Layer Test to Catch It Before Handoff

After that incident, tests/test_windows_cp932_safety.py was written specifically to catch this class of bug on macOS — before code targeting Windows ever leaves the development machine.

1. A static check that walks every character. Scripts that get subprocess-executed or output-captured during the build/release pipeline are listed in CP932_CRITICAL_SCRIPTS, and every character in those files is checked for cp932-encodability:

for ch in line:
    try:
        ch.encode("cp932")
    except UnicodeEncodeError:
        problems.append(f"{rel}:{lineno} '{ch}' cannot be encoded as cp932")

This flags the offending character without ever running the script, and the fix is simple: swap emoji for ASCII markers like [OK] or [NG].

2. A behavioral check that reproduces cp932 stdout for real. The static check alone can't confirm the script actually crashes at runtime, so a second test runs bump_version.py --check as a real subprocess with the environment variables PYTHONIOENCODING=cp932 and PYTHONUTF8=0 set:

env = {**os.environ, "PYTHONIOENCODING": "cp932", "PYTHONUTF8": "0"}
r = subprocess.run(
    [sys.executable, "tools/bump_version.py", "--check"],
    cwd=ROOT, capture_output=True, env=env,
)

PYTHONIOENCODING forces Python's standard streams to use a specific encoding, and PYTHONUTF8=0 disables the automatic UTF-8 mode that newer Python versions otherwise enable by default. Together, these let a macOS process convincingly simulate what a Japanese-Windows console would actually do — catching runtime behavior that a static character check alone can't confirm.

Takeaway

Encoding mismatches tend to hide the longer a team's development machines default to UTF-8. Any code path that captures subprocess output or moves text through a pipe inherits the encoding of the environment it runs in unless told otherwise, which leaves a blind spot no amount of testing on a UTF-8 machine will ever expose. Japanese text itself isn't the risk — cp932 handles it fine. The risk is anything outside cp932's table, emoji being the most common example, slipping into output on a Japanese-Windows machine. The fix has two parts: keep emoji out of anything that runs cross-platform, and where that's not practical, write a test that deliberately reproduces the target locale's encoding rather than trusting that a passing test on your own machine means anything about someone else's.