48-Hour Field Notes: The Writer Was Fine. Four Process Defaults Had Already Voted.

I spent the first evening blaming a tiny report writer, because the artifact looked wrong only on the second host. Locally the created file landed at mode 0644, and the sidecar clock string matched my laptop zone. On the clean run the mode was 0664, and the same instant wore a different UTC offset. Would you have opened the serializer first, or would you have printed the process defaults before touching a single line?

The first twelve hours were a reading exercise

I kept rereading the function that opened the report, because a wrong mode feels like a bug hiding in the open call. The call used the default creation mode, which means the process umask still gets the last vote. I also kept rereading the timestamp helper, because the rendered string was not the epoch I thought I had stored. Have you ever lost half a day inside a function that was doing exactly what the runtime asked?

I added prints around the writer, and both hosts still disagreed after those prints looked identical. The bytes of the report body matched, so this was not a drifted request and it was not a locale mismatch either. The disagreement lived in the file mode, the zone name, one temp path, and an unordered token. Why would four quiet defaults move together when the source file had not moved at all?

What I printed before I trusted either host

I stopped reading the writer and started printing the process, because the file was only the last visible symptom. A fresh Python process inherits umask, TZ, TMPDIR, and PYTHONHASHSEED from the parent that actually launched it. My editor terminal and the clean job were not the same parent, even though both claimed to run the same script. If the parent is different, why would I expect the child defaults to match without a written record?

These notes assume a current CPython 3 on a POSIX host, and the script below prints sys.version so the record is whatever you actually ran. I am not pinning a patch release from memory, and I am not treating Windows permission bits as the same story. The behavior I relied on is the documented contract for os.umask, TZ, tempfile.gettempdir, and PYTHONHASHSEED, which is worth rereading beside the os, environment, tempfile, and datetime pages before you freeze a test.

The mode was a vote, not a constant

Python asks for 0o666 on a normal new file, and the kernel then applies the complement of the current umask. A laptop umask of 0o022 yields 0o644, while a umask of 0o002 yields 0o664, and neither result is a lie. I had asserted an exact mode in a snapshot test, which froze one host's habit as if it were the file format. Would you call that a portability bug, or would you call it an input I forgot to write down?

Passing 0o640 to os.open does not skip that vote, because the creation mode is still masked. If the contract really needs those bits, I create the file and then chmod, and I keep that file out of a shared directory. There is a short race between create and chmod, so this is a report-file fix, not a recipe for secrets.

The clock string was a second input

The epoch seconds matched once I printed time.time, but datetime.now() without a timezone still formatted local wall time. One process had TZ unset and followed the host zone, while the other had TZ=UTC in the job environment. I already record locale before I blame a fixture, so I wrongly assumed the clock was settled too. Does a matching epoch mean the rendered string is safe, or only that the instant was the same?

The token moved because a set is not a list

A short token in the sidecar came from a set of header names, and I had joined that set without sorting it. Set order follows hash randomization, so an unset PYTHONHASHSEED can change that joined string across processes. I had not set the variable, and I had not sorted the names, so both omissions were mine. If the token is part of the contract, why was its order left to the allocator?

Builtin hash() of a string is a per-process diagnostic, not an identifier I should store. Hash randomization is also a mitigation against hash-flooding, so freezing the seed in production would be the wrong kind of stability. Sort the names, or hash the sorted bytes with hashlib, and leave the seed alone unless you are debugging.

The script I would run before the next blame

This is the check I wish I had run at hour one. It restores the umask after reading it, writes a probe file, and prints both the moving fields and the fields I am willing to diff. It does not export PYTHONHASHSEED, and it does not claim that two hosts should match on purpose-built moving values.

#!/usr/bin/env python3
"""Print process defaults that vote after the source already matches."""

from __future__ import annotations

import hashlib
import json
import os
import stat
import sys
import tempfile
import time
from datetime import datetime, timezone
from pathlib import Path


def current_umask() -> int:
    previous = os.umask(0o022)
    os.umask(previous)  # restore; the probe must not change the process
    return previous


def sample_file_mode(directory: Path) -> str:
    path = directory / "mode-probe.txt"
    flags = os.O_CREAT | os.O_WRONLY | os.O_TRUNC
    fd = os.open(path, flags, 0o666)
    try:
        os.write(fd, b"probe\n")
    finally:
        os.close(fd)
    mode = oct(stat.S_IMODE(path.stat().st_mode))
    path.unlink()
    return mode


def main() -> None:
    names = ["X-Request-Id", "X-Trace", "Accept"]
    unordered = ",".join(set(names))
    ordered = ",".join(sorted(set(names)))
    fingerprint = {
        "python": sys.version.split()[0],
        "platform": sys.platform,
        "umask": oct(current_umask()),
        "tz_env": os.environ.get("TZ"),
        "tzname": list(time.tzname),
        "naive_local": datetime.now().isoformat(timespec="seconds"),
        "aware_utc": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "tmpdir_env": os.environ.get("TMPDIR"),
        "tempdir": tempfile.gettempdir(),
        "pythonhashseed_set": "PYTHONHASHSEED" in os.environ,
        "sample_str_hash": hash("field-note"),
        "unordered_token": unordered,
        "sorted_token": ordered,
        "stable_digest": hashlib.sha256(ordered.encode()).hexdigest()[:12],
    }
    with tempfile.TemporaryDirectory(prefix="proc-defaults-") as raw:
        directory = Path(raw)
        fingerprint["sample_file_mode"] = sample_file_mode(directory)
        fingerprint["probe_dir"] = raw
    json.dump(fingerprint, sys.stdout, indent=2, sort_keys=True)
    sys.stdout.write("\n")


if __name__ == "__main__":
    main()

I saved that as proc_defaults.py and treated the JSON as a lab notebook page, not as a golden file. The compare step ignores fields that are allowed to move, then shouts about the rest.

python3 proc_defaults.py > local.json
# copy the same script to the clean job, then:
python3 proc_defaults.py > remote.json
python3 - <<'PY'
import json
left = json.load(open("local.json"))
right = json.load(open("remote.json"))
ignore = {
    "naive_local", "probe_dir", "sample_str_hash",
    "unordered_token", "tempdir", "aware_utc",
}
for key in sorted(set(left) | set(right)):
    if key in ignore:
        continue
    if left.get(key) != right.get(key):
        print(f"DIFF {key}: {left.get(key)!r} != {right.get(key)!r}")
PY

aware_utc is in the ignore list only because the two runs are not the same second. If those timestamps differ by hours, I would still read tz_env before I touch the formatter.

What I am willing to diff

Field	If it differs	What I would repeat
umask, sample_file_mode	A snapshot of mode bits fails	Record the mask, and chmod only when the contract names the bits
tz_env, tzname, naive_local	The sidecar clock string drifts	Store an aware UTC value, and treat TZ as an input
unordered_token, sample_str_hash	A token or builtin hash moves	Sort before joining, and do not store builtin hash()
tempdir, probe_dir	A path snapshot fails	Do not assert temp paths; record TMPDIR as context only
python, platform	Behavior shifts under you	Pin the runtime in the job spec, not in a paragraph

A second process, not a bigger story

I did not need a bigger harness, and I did not need a named model with a quoted score. I needed a skeptical pass over the checklist, plus one process that was not my editor terminal. MonkeyCode's free model access and its free server option were enough for that narrow pass, and I am not claiming more than that.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I asked the free model to challenge the checklist, not to invent a quota, a hardware shape, or a promise that the option would last. Then I ran the same fingerprint on the free server, so the second printout came from a clean process rather than from my laptop shell. The useful part was the diff of those two printouts, not a leaderboard and not a story about speed. If a default is missing from the printout, can a second host even tell you that it moved?

What the probe itself broke

The first version called os.umask and forgot to restore the previous mask, so the probe changed the process it claimed only to observe. A later version printed tempfile.gettempdir() without also printing TMPDIR, which hid an unset variable whose platform default still differed. I also printed hash() of a sample string, then remembered that a fixed seed is only a debug lever. It is not a production setting that I should silently export into every clean job I run. Have you ever shipped the probe and quietly changed the thing you meant only to observe?

I also put probe_dir into the first snapshot, and of course that path changed on every run. A temporary directory is supposed to move, so asserting it was me asking the wrong contract. The hours after that were less dramatic: drop the moving fields from the diff, keep them in the notebook, and stop rewriting a writer that had been fine.

What I would repeat on the next mismatch

1. I would print umask, TZ, TMPDIR, and whether PYTHONHASHSEED is set before I open the writer again.
2. I would store an aware UTC timestamp, and I would keep the naive local string only as a human clue.
3. I would sort every set before it becomes a token, and I would use hashlib if I need a stable digest.
4. I would chmod after create only when a named mode is the contract, and I would not do that for secrets.
5. I would diff two printouts from two parents, then ignore fields that are designed to move.
6. I would restore every process-global knob the probe touches, starting with umask, before I trust the page.

The repeatable artifact is the script plus that ignore list, not a memory of which host felt normal. A field note that cannot be rerun is just a mood.

Who should leave this notebook closed

This approach is a small reproducibility check, and it is a poor fit for several jobs I would not blur into it.

• Do not send secrets, customer data, or production credentials to a free server just to borrow a second process. A free process is not a trust boundary, a private network, or a compliance lab.
• Do not use the mode dance as a Windows permission test. The vote I described is POSIX-shaped, and the script's mode field will not teach you an ACL.
• Do not disable hash randomization in production so logs look stable. Sort the data instead, and keep the flood mitigation.
• Do not treat free model access or a free server as a quota, a hardware spec, or a permanent offer. I was not given those facts, so I am not writing them.
• Do not expect this page to catch locale, encoding, shell identity, or the working directory. Those are neighboring failures, and this note is only the defaults that vote after the source matches.

os.umask is also process-global, so a threaded test can race if another thread changes it while you read. Naive local timestamps stay ambiguous across a daylight-saving fold, even after you print TZ. I did not measure runtime, and I am not ranking any model.

If the next mismatch shows up only on a clean job, I would borrow a second process and a skeptical reader before I rewrite the writer. The notebook still has to name the defaults, or the second process just gives you a second mystery.