PowerShell on Legacy Chinese Windows: UTF-8 with BOM or Bust

Windows PowerShell 5.1 does not guess encodings. When it loads a .ps1 file, it looks for a byte-order mark. If it finds one, it decodes accordingly. If it does not, it falls back to the system's ANSI code page — and on a Chinese Windows install, that code page is GBK (CP936). Every non-ASCII character in your script is then decoded through the wrong table, and you get mojibake that does not throw an error. It just quietly changes what your script means.

This is the story of how a path check turned into a week of debugging, and the one rule that fixed it: every .ps1 file ships as UTF-8 with BOM and CRLF endings.

The symptom

A deployment script had a line like this:

$configDir = Join-Path $env:ProgramData "我的工具\配置"
if (-not (Test-Path $configDir)) {
    New-Item -ItemType Directory -Path $configDir -Force | Out-Null
}

On the developer's machine it worked. On the build server it created a directory named something like 我的工具 — or sometimes it created nothing at all, and Test-Path returned $false for a path that clearly existed when inspected manually in Explorer.

The reason is straightforward once you see the bytes. The file was saved by an editor as UTF-8 without BOM. PowerShell 5.1 read it as GBK. The UTF-8 bytes for 我 are E6 88 91. Decoded as GBK, those three bytes are a valid two-byte character plus a stray byte, producing a different string entirely. PowerShell then passed that corrupted string to the Windows API, which dutifully created a directory with a nonsense name. Test-Path on the intended name failed because the intended name was never used.

No exception. No warning. Just a different program than the one you wrote.

Detecting it: hex-dump the first bytes

The fastest way to confirm the diagnosis is to look at the raw bytes at the start of the file.

# Show the first 8 bytes of a script file
$bytes = [System.IO.File]::ReadAllBytes(".\deploy.ps1")[0..7]
($bytes | ForEach-Object { $_.ToString("X2") }) -join ' '

Interpretation:

• EF BB BF — UTF-8 with BOM. PowerShell 5.1 decodes correctly.
• FF FE — UTF-16 LE with BOM. Also fine.
• FE FF — UTF-16 BE with BOM. Fine, rare.
• Anything else — no BOM. PowerShell 5.1 falls back to the ANSI code page. On a GBK system that means GBK.

If you see E6 88 91 at the start of a file containing 我, you have a BOM-less UTF-8 file, and 5.1 is about to misread it.

A quick Python check across a whole directory is often easier than doing this by hand:

from pathlib import Path

BOMS = {
    b"\xef\xbb\xbf": "utf-8-sig",
    b"\xff\xfe": "utf-16-le",
    b"\xfe\xff": "utf-16-be",
}

def sniff(path: Path) -> str:
    head = path.read_bytes()[:4]
    for bom, enc in BOMS.items():
        if head.startswith(bom):
            return enc
    # No BOM. Try strict UTF-8; if it fails, it is almost certainly ANSI/GBK.
    try:
        path.read_bytes().decode("utf-8")
        return "utf-8 (NO BOM) <-- 5.1 will misread this"
    except UnicodeDecodeError:
        return "ansi/gbk (NO BOM)"

for ps1 in Path("scripts").rglob("*.ps1"):
    print(f"{ps1}: {sniff(ps1)}")

Run this on any legacy repo and the offending files surface immediately. On a Chinese Windows box, the mix of "utf-8 (NO BOM)" and "ansi/gbk" files is the whole problem in one listing.

Why BOM-less UTF-8 is a trap on Chinese Windows

The default encoding for reading files in PowerShell 5.1 is not UTF-8. It is [System.Text.Encoding]::Default, which maps to the active ANSI code page. On an English system that is CP1252. On a Simplified Chinese system it is CP936 (GBK). On a Japanese system, CP932. The behavior of the same script file therefore depends on the locale of the machine running it.

This is the part that makes the bug so expensive. The script is not wrong on every machine — it is wrong on machines whose code page differs from the one the author implicitly assumed. A BOM removes the ambiguity: it tells the parser "this is UTF-8" regardless of locale.

The same trap applies to Get-Content and Set-Content in 5.1. Get-Content without -Encoding uses the ANSI default; Set-Content without -Encoding writes ANSI. Round-tripping a UTF-8 file through those cmdlets silently converts it to GBK.

Bulk-converting a repo to UTF-8 with BOM

You can fix a directory of scripts with PowerShell itself. The trick is to read the bytes, decode them with the correct source encoding, and write them back with a BOM.

# convert-to-utf8bom.ps1
param(
    [Parameter(Mandatory)][string]$Root,
    [string]$SourceEncoding = "utf-8"   # use "gb2312" for legacy GBK files
)

$utf8Bom = New-Object System.Text.UTF8Encoding($true)

Get-ChildItem -Path $Root -Recurse -Filter *.ps1 | ForEach-Object {
    $path  = $_.FullName
    $bytes = [System.IO.File]::ReadAllBytes($path)

    # Already has a UTF-8 BOM? Skip.
    if ($bytes.Length -ge 3 -and $bytes[0] -eq 0xEF -and $bytes[1] -eq 0xBB -and $bytes[2] -eq 0xBF) {
        Write-Host "skip (already BOM): $path"
        return
    }

    $text = [System.Text.Encoding]::GetEncoding($SourceEncoding).GetString($bytes)

    # Normalize line endings to CRLF.
    $text = $text -replace "`r`n", "`n" -replace "`n", "`r`n"

    [System.IO.File]::WriteAllText($path, $text, $utf8Bom)
    Write-Host "converted: $path"
}

Two things matter here. First, WriteAllText with a UTF8Encoding($true) instance emits the BOM; UTF8Encoding($false) does not. Second, you must know the source encoding. If a file is genuinely GBK, decoding it as UTF-8 will throw or produce replacement characters. The Python sniffer above tells you which is which before you run the conversion.

The same job in Python, which is often more convenient in CI:

import sys
from pathlib import Path

def to_utf8_bom(path: Path, source_encoding: str) -> None:
    raw = path.read_bytes()
    if raw.startswith(b"\xef\xbb\xbf"):
        return
    text = raw.decode(source_encoding)
    text = text.replace("\r\n", "\n").replace("\n", "\r\n")
    path.write_bytes(b"\xef\xbb\xbf" + text.encode("utf-8"))

for p in Path(sys.argv[1]).rglob("*.ps1"):
    try:
        to_utf8_bom(p, "utf-8")
    except UnicodeDecodeError:
        to_utf8_bom(p, "gbk")
    print(f"converted {p}")

The fallback to gbk on UnicodeDecodeError handles mixed repos where some files were saved as ANSI by an older editor. Once every file is UTF-8 with BOM, the ambiguity is gone.

PowerShell 7 behaves differently

PowerShell 7 (Core) changed the defaults. Get-Content and the script loader default to UTF-8 without requiring a BOM, and Set-Content writes UTF-8 without BOM by default. So the same BOM-less file that 5.1 mangles will load correctly in 7.

That is a feature, and also a trap. If your scripts run under both 5.1 and 7 — which is common on servers that still have Windows PowerShell installed alongside Core — you need the file to work under the stricter reader. UTF-8 with BOM works in both. BOM-less UTF-8 works only in 7. So the rule is not "use UTF-8"; the rule is "use UTF-8 with BOM" if any 5.1 host might touch the file.

Note also that PowerShell 7 still honors the BOM when present, so adding it does not break anything under Core.

A pre-flight check before you ship

The fix is cheap; the discipline is what fails. Put a validator in your build or pre-commit step so a BOM-less file never reaches a 5.1 host.

#!/usr/bin/env python3
"""Fail the build if any .ps1 is not UTF-8 with BOM and CRLF."""
import sys
from pathlib import Path

def check(path: Path) -> list[str]:
    errors = []
    raw = path.read_bytes()

    if not raw.startswith(b"\xef\xbb\xbf"):
        errors.append("missing UTF-8 BOM")
    else:
        try:
            raw[3:].decode("utf-8")
        except UnicodeDecodeError as exc:
            errors.append(f"not valid UTF-8: {exc}")

    # CRLF check: reject lone LF outside of the final byte.
    body = raw[3:] if raw.startswith(b"\xef\xbb\xbf") else raw
    if b"\n" in body.replace(b"\r\n", b""):
        errors.append("contains LF-only line endings; expected CRLF")

    return errors

failed = False
for p in sorted(Path(".").rglob("*.ps1")):
    errs = check(p)
    if errs:
        failed = True
        for e in errs:
            print(f"{p}: {e}")

sys.exit(1 if failed else 0)

Drop this into CI as a gate. The failure message is specific enough that the fix is obvious: save the file as UTF-8 with BOM, convert line endings to CRLF, commit.

If you want the validator to auto-fix instead of fail, reuse the conversion functions above and re-run the check. In a pre-commit hook, auto-fixing is usually the better default — it removes the human from the loop entirely.

The rule

Every .ps1 file that might ever run under Windows PowerShell 5.1 must be saved as UTF-8 with BOM and CRLF line endings. BOM-less UTF-8 is a valid encoding and a broken script on a GBK system. UTF-16 also works but is unusual in a repo and confuses diffs and tooling. Stick to UTF-8 with BOM.

The failure mode — silent mojibake, locale-dependent behavior, no exception — is what makes this worth a hard rule rather than a convention. A linter catches it in a second; a human catches it after a week of Test-Path returning $false on a path that exists.

More notes like this ship every week on this site.