48-Hour Field Notes: The Log Was UTF-8. open() Still Used ASCII.

Have you ever shipped a log parser that only failed after you left your laptop behind? Mine chewed through weeks of UTF-8 logs at home, then exploded on the first remote run. The file looked identical on disk, and the traceback pointed at a bare open() call. No encoding argument sat on that call, and the UnicodeDecodeError never appeared on my laptop.

I almost blamed stdout again, because a silent TTY had burned me earlier this month. This time isatty() returned true on my interactive shell, and the file still decoded on the laptop. The remote failure lived in a systemd unit and a cron slot, where LANG was empty. Empty locale on that box did not mean UTF-8. It meant ASCII.

Hour 0–6: I blamed the file

The first hours went to the log itself, because that is the comfortable suspect. I copied the same bytes home with scp, then ran file, hexdump, and a tiny Python one-liner. Every check said UTF-8. Every laptop parse succeeded. So why did the service unit die on the same path?

Commands I actually ran, in this order:

file /var/log/app/app.log
hexdump -C /var/log/app/app.log | head
sha256sum /var/log/app/app.log ~/Downloads/app.log
python3 -c "print(open('/var/log/app/app.log').read()[:80])"

file reported UTF-8 text. hexdump showed the café bytes c3 a9, not a Windows c9. The checksums matched across machines. The one-liner printed the header on my laptop and raised UnicodeDecodeError under cron. Same inode content. Different decoder.

Was the log truncated on the server? No. Was there a BOM I had missed? Also no. I even checked for CRLF, because that family of bugs still lives in my muscle memory, and this file used plain \n.

Hour 6–18: I blamed Python

Next I convinced myself the interpreter was lying about encodings. I printed sys.getdefaultencoding() on both machines and got utf-8 twice. That result felt reassuring, and it was the wrong reassurance. Default encoding is the Unicode internals codec, not the codec open() picks for text files.

The call that actually matters is locale.getpreferredencoding(False). On the laptop it returned UTF-8. Under the systemd unit it returned ANSI_X3.4-1968, which is ASCII with a formal name. sys.flags.utf8_mode was 0 in both places until I set PYTHONUTF8 myself.

This is the probe I wish I had run at hour one. Save it as probe_locale.py and run it in the same environment as the parser, not in an interactive shell that already exported LANG.

#!/usr/bin/env python3
"""Locale encoding probe. Run it as the service user, not in your laptop shell."""
from __future__ import annotations

import locale
import os
import sys
import tempfile
from pathlib import Path

SAMPLE = "ERROR café 温度 🔥\n"


def main() -> None:
    preferred = locale.getpreferredencoding(False)
    stdout_enc = getattr(sys.stdout, "encoding", None)
    print(f"sys.version               = {sys.version.split()[0]}")
    print(f"sys.flags.utf8_mode       = {sys.flags.utf8_mode}")
    print(f"sys.getdefaultencoding()  = {sys.getdefaultencoding()!r}")
    print(f"preferred encoding        = {preferred!r}")
    print(f"sys.stdout.encoding       = {stdout_enc!r}")
    for key in ("LANG", "LC_ALL", "LC_CTYPE", "PYTHONUTF8"):
        print(f"{key:24} = {os.environ.get(key)!r}")

    with tempfile.TemporaryDirectory() as tmp:
        path = Path(tmp) / "app.log"
        path.write_bytes(SAMPLE.encode("utf-8"))
        try:
            text = path.read_text()  # locale encoding on purpose
            print(f"read_text() default       = {text!r}")
        except UnicodeDecodeError as exc:
            print(f"read_text() default FAILED: {exc}")
        print(f"read_text(encoding=utf-8) = {path.read_text(encoding='utf-8')!r}")

    try:
        sys.stdout.write(SAMPLE)
    except UnicodeEncodeError as exc:
        print(f"stdout write FAILED: {exc}")


if __name__ == "__main__":
    main()

Reproduce the cron environment without guessing:

# Interactive laptop shell: this usually passes.
python3 probe_locale.py

# Cron / empty locale: this is the failure I needed.
env -i PATH="$PATH" HOME="$HOME" LANG=C LC_ALL=C python3 probe_locale.py

# Process-wide UTF-8 mode, still with LANG=C.
env -i PATH="$PATH" HOME="$HOME" LANG=C LC_ALL=C PYTHONUTF8=1 python3 probe_locale.py

# Surface accidental open() calls on Python 3.10+.
PYTHONWARNDEFAULTENCODING=1 python3 -W default::EncodingWarning probe_locale.py

The env -i run is the whole story. read_text() without encoding= raises. read_text(encoding="utf-8") returns the café line. sys.getdefaultencoding() still says utf-8, which is why that printout wasted six hours.

Hour 18–36: I blamed the container, then the unit file

I wasted a stretch on the image, because slim Python images used to ship a C locale. Official images have set LANG=C.UTF-8 for years, so that theory was stale on arrival. The unit file was the actual hole. It listed ExecStart and a User, and it listed no Environment= lines for locale. Cron was worse: a stripped environment, no PAM session, no LANG.

Does your service inherit the locale from your SSH session? Mine did not. I logged in, ran the parser by hand, and watched it pass. Then systemctl start failed again. That gap is the bug. Interactive shells lie.

I needed a machine whose login locale would not hide the unit file. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access and free server option to iterate on the probe under an empty LANG, because my laptop cannot produce ASCII open() behavior no matter how often I restart the shell.

The model did not invent a root cause I had missed. It did keep the probe script honest while I compared read_text() against open(..., encoding="utf-8") on that box. I still had to run systemctl show-environment myself.

systemctl show-environment
systemctl cat app-parser.service
crontab -l
# Simulate the unit without waiting for the timer.
sudo systemd-run --uid=app --wait --pipe \
  -p Environment=LANG=C -p Environment=LC_ALL=C \
  /usr/bin/python3 /opt/app/probe_locale.py

A small decision table

I wanted one page I could paste into the pull request. This is that page.

Lever	Fixes open() / Path.read_text()	Fixes print() to a pipe	Survives cron / systemd	Cost
encoding="utf-8" on every text open()	Yes	No	Yes	Code change, the one I trust
PYTHONUTF8=1 in the unit	Yes, for that process	Yes	Only if the unit sets it	One environment line
LANG=C.UTF-8 in the unit	Usually	Usually	Only if libc has UTF-8 locales	Depends on the image
locale-gen plus a full en_US.UTF-8	Maybe	Maybe	Image-specific	Too much for a parser
Trust sys.getdefaultencoding()	No	No	No	This is how I lost a day

Numbered rules I will keep next time:

1. Pin encoding="utf-8" on every text file the parser owns, including temp files.
2. Set PYTHONUTF8=1 in the systemd unit as a belt, not as the only buckle.
3. Run the probe with env -i and with systemd-run, never only in SSH.
4. Turn on EncodingWarning in CI so a bare open() fails the build.

A minimal parser change looks like this. The old line is the bug. The new line is the fix.

from pathlib import Path

def load_log(path: Path) -> str:
    # Bug: locale encoding. ASCII under cron, UTF-8 on my laptop.
    # return path.read_text()
    return path.read_text(encoding="utf-8")

And the unit fragment I ended up with:

[Service]
User=app
Environment=PYTHONUTF8=1
Environment=PYTHONWARNDEFAULTENCODING=1
ExecStart=/usr/bin/python3 /opt/app/parse_logs.py

What broke while I was “fixing” it

PYTHONUTF8=1 fixed file reads and then surprised me on a binary sidecar. One helper used open(path, "r") on a gzip member that was not text. UTF-8 mode still wants a decode, so that helper started raising. I switched that path to "rb" and left UTF-8 mode enabled. Text logs are text. Gzip is bytes. Mixing those in one function is how locale bugs hide.

Stdout was a second trap. After file reads succeeded, a debug print of the café line failed when cron piped output into logger. sys.stdout.encoding was ASCII in that pipe. print() is not a logfile API. I wrote the same line with encoding="utf-8" into a file instead.

I also tried locale.setlocale(locale.LC_ALL, "C.UTF-8") inside the process. It worked on the box that had that locale compiled, and it failed on the box that did not. Application code should not be a locale installer. Pin the codec. Do not negotiate it.

What I would repeat

I would repeat the probe before I repeat the theory. Forty-eight hours is a long time to spend on a missing keyword argument. The next parser I touch gets encoding="utf-8" on day zero, plus an env -i job in CI that feeds a café line through the parser.

Would I debug this only on a laptop again? No. Laptops lie about locale the same way they lie about time zones, spawn methods, and IPv6. The failing environment is the one whose LANG is empty. Reproduce there, even if that environment is a free server you throw away after the probe.

A tiny CI check that would have shortened this:

# Label: example snippet for a job, not a full pipeline.
- name: Parser must survive ASCII locale
  run: |
    env -i PATH="$PATH" LANG=C LC_ALL=C \
      python3 -c "from parse_logs import load_log; from pathlib import Path;
      p = Path('tests/fixtures/cafe.log'); print(load_log(p)[:32])"

If that job fails, you have the bug. If it only fails on your laptop, you do not have the bug yet.

Limitations, and who should skip this

This workflow is for text logs you own, on Linux services whose locale you do not fully control. It is not a license to guess encodings for binary formats, vendor exports, or mixed Windows code pages. If your codebase already passes encoding= everywhere and your unit files already set PYTHONUTF8, you do not need a 48-hour tour.

Do not use a coding model as a substitute for locale.getpreferredencoding(False). The model can draft the probe. It cannot see your systemd environment unless you paste systemctl show-environment. Do not set PYTHONUTF8=1 globally on a host that still reads legacy 8-bit files on purpose. And do not treat C.UTF-8 as universal; some images simply do not ship that locale.

I am also not claiming a benchmark, a quota, or a forever-free box. The useful part of this writeup survives if you run the probe on any VM whose LANG is empty. The café fixture is the artifact. The rest is commentary.

If you have a spare environment that is not UTF-8, run probe_locale.py there before you trust a log parser. That is the whole habit I am keeping.