The UTF-8 / Code Page Surprise That Corrupted My Log Parser

The parser wasn’t wrong. The bytes were fine. My shell lied about what “text” meant—and I shipped a log tool that mangled non-ASCII lines until a Czech filename in a support zip made the bug undeniable.

This is a diagnostic post about Windows code pages, PowerShell encoding defaults, and why “just use UTF-8” is a process, not a slogan.

The failure symptom

A small PowerShell script:

1. Ran an external tool that emitted logs.
2. Captured output / read a .log file.
3. Parsed timestamps and paths with regex.
4. Wrote a summary CSV.

On my machine: green. On a colleague’s machine: paths with diacritics became ?? or mojibake, and the CSV “couldn’t find” files that Explorer showed clearly.

Classic encoding heisenbug: locale + shell + redirect + tool disagreed.

The moving parts (Windows edition)

• System ANSI code page (legacy) still influences some apps and CMD behaviors.
• OEM code page shows up in consoles historically (chcp).
• PowerShell 5.1 vs PowerShell 7 differ in how often UTF-8 saves you by default.
• Redirects and pipelines to native executables can re-encode.
• File reads without -Encoding use defaults you didn’t consciously pick.

If you only test with ASCII, you never meet these parts.

The diagnostic sequence that found it

1. Look at raw bytes, not the rendered console string. A hex glance (or reading as byte array) showed UTF-8 sequences that the parser later interpreted as something else.
2. Check the console code page: chcp in CMD; in PowerShell, inspect [Console]::OutputEncoding and $OutputEncoding.
3. Check how the file was read/written: Get-Content / Set-Content / Out-File encoding parameters.
4. Ask which PowerShell edition ran in CI vs laptop.
5. Reproduce with a non-ASCII fixture checked into the repo (the missing test).

The smoking gun was a 5.1 path writing a file with a default that wasn’t UTF-8, then a later step reading it as UTF-8 (or vice versa). ASCII tests stayed green.

Fixes that are boring and correct

• Pin encoding at every boundary: Get-Content -Encoding utf8, Export-Csv -Encoding utf8, etc. (know when utf8BOM matters for Excel-adjacent humans).
• Prefer PowerShell 7 where allowed for friendlier defaults—but still pin.
• For CMD-era tools, know chcp 65001 is a blunt instrument with side effects; don’t sprinkle it as superstition.
• Add a CI fixture: Contrôle-Über-škoda.log or similar. If CI can’t round-trip it, you don’t ship.

Teaching takeaway

Encoding is not an advanced elective. It’s a diagnostic literacy topic that belongs next to PATH and exit codes—especially for Windows learners who live across CMD and PowerShell.

Minimal reproduction you can keep in a repo

1. Create a file with a non-ASCII name and UTF-8 contents.
2. Read/write it through your script on PowerShell 5.1 and 7.
3. Compare hashes / byte lengths.
4. Fail CI if they diverge from the expected encoding.

If that sounds heavy for a “small parser,” remember: the corruption only shows up for the customers you care about most—the ones whose languages don’t fit in seven-bit nostalgia.

Console rendering ≠ file encoding

Another trap: the console can display substitutions while the file on disk is fine (or the reverse). Always pin the layer you’re judging—screen, pipeline, or bytes on disk.

Empathy note for global teams

If your tooling assumes ASCII filenames, you’re quietly failing colleagues and customers whose names, cities, and documents don’t. Encoding literacy is inclusion work wearing debugging clothes.

I now treat “non-ASCII path survives the pipeline” as a required drill, not a niche. Interactive lessons in CMD Master are a natural place to force that fixture early, before someone corrupts a customer export.

If your parser only ever saw ASCII, you don’t have a parser. You have a happy path.