I Trusted os.replace for 48 Hours. The Dest Path Lived on Another Mount.

Have you ever watched a consumer throw JSONDecodeError and immediately blamed the process that wrote the file? I did, and I kept staring at flush calls because the writer already used os.replace. My laptop never reproduced the torn file, which made the outage feel cursed and strangely personal. The real cause was hiding in mount output, not in the json module and not in disk caching.

This is a 48-hour field note, not a postmortem with fake graphs. I will tell you what I tried, what broke under me, and what I would run again tomorrow. You can reproduce the whole mess on any POSIX box whose /tmp is not the same device as your config directory.

What I thought was true

On POSIX, rename is atomic when the source and destination live on the same filesystem. Python's os.replace is the usual way to expose that guarantee without a half-written visible file. So if a reader saw truncated JSON, the writer must have skipped fsync, or the volume was lying. That story is comforting because it keeps the bug inside your process, where logging can still reach it.

I also trusted a helper that looked responsible in review. It wrote a tempfile, flushed, and then "atomically" moved the tempfile onto the live path. Does that sentence already make you nervous? It should, because the tempfile directory and the live path are not required to share a device.

Field notes from the 48 hours

Hours 0–8: I chased flush, then I chased NFS

The consumer logged JSONDecodeError a few times an hour, always against the shared config file. I added flush, then os.fsync, then a cowardly sleep, because that is what you do when you panic. Nothing changed, which should have been the clue that the write path was not staying on one device. Was the volume lying to me? I started blaming NFS attribute cache like everyone does under pressure.

Commands I actually ran, in roughly this order:

# Is the writer even the process I think it is?
ps aux | grep config-writer

# Does the live file look truncated at the moment of the error?
stat /var/app/config.json
xxd /var/app/config.json | tail

# Classic wrong turn: blame the network filesystem.
mount | grep /var
nfsstat -c 2>/dev/null || true

The file was sometimes valid JSON and sometimes a prefix of JSON. That pattern is a torn read, not a permissions problem, and not a pretty-printer going rogue.

Hours 8–24: strace told on the copy

I ran strace against the writer and expected rename, because that is what os.replace should issue. Instead I saw open, read, write, and unlink, which is a copy, not a swap. Why would Python copy a config file that I had asked it to replace in place, atomically?

# Attach to the writer. Filter the syscalls that matter.
sudo strace -f -p "$WRITER_PID" -e trace=openat,rename,renameat,renameat2,read,write,unlink,fsync

The stack was not calling os.replace on the live path. A convenience helper used tempfile.NamedTemporaryFile in the default directory, then shutil.move onto /var/app/config.json. shutil.move tries os.rename first, catches EXDEV, and copies. Copy is visible. Copy can be read halfway. Copy is how a "safe replace" becomes a torn file.

I printed errno in a tiny probe after that, because I wanted the kernel's words, not my story.

# labeled demonstration: probe whether two paths share a device
import os
import tempfile

src_dir = tempfile.gettempdir()
dst_dir = "/var/app"
print("tmp", src_dir, os.stat(src_dir).st_dev)
print("dst", dst_dir, os.stat(dst_dir).st_dev)
print("same device?", os.stat(src_dir).st_dev == os.stat(dst_dir).st_dev)

When st_dev differs, os.replace will not save you. It raises OSError with errno.EXDEV. If your helper catches that and falls back to copy, you silently gave up atomicity.

Hours 24–48: the laptop could not lie the same way

My laptop keeps /tmp on the same root filesystem as the app directory, so replace always worked. The box that served traffic had /tmp on tmpfs, and the config directory lived on a separate disk. Same Python, same git sha, different mount table, different failure mode. How often do we even check that before we ship a "safe write" helper?

findmnt -T /tmp
findmnt -T /var/app
stat -c '%d %n' /tmp /var/app

I needed a second Linux environment that did not quietly inherit my laptop's single-disk layout at all. I used MonkeyCode's free server option for that check, plus free model access to draft the first repro script. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you already have a spare VM, use that instead; the lesson is the mount table, not the vendor.

The artifact: a same-directory swap you can run

The fix is boring, which is why it keeps getting skipped. Create the tempfile in the destination directory, fsync the bytes, then os.replace onto the live name. Do not catch EXDEV and "helpfully" copy unless you have another concurrency story.

# labeled demonstration: POSIX same-directory replace
# Run it on a machine where /tmp and the dest dir may differ.
from __future__ import annotations

import errno
import json
import os
import stat
import tempfile
from pathlib import Path
from typing import Any


def same_device(a: str | os.PathLike[str], b: str | os.PathLike[str]) -> bool:
    return os.stat(a).st_dev == os.stat(b).st_dev


def atomic_write_json(dest: Path, payload: Any) -> None:
    dest = dest.resolve()
    dest.parent.mkdir(parents=True, exist_ok=True)

    fd, tmp_name = tempfile.mkstemp(
        dir=str(dest.parent),
        prefix=f".{dest.name}.",
        suffix=".tmp",
    )
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as handle:
            json.dump(payload, handle, separators=(",", ":"), sort_keys=True)
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(tmp_name, dest)
    except OSError as exc:
        try:
            os.unlink(tmp_name)
        except FileNotFoundError:
            pass
        if exc.errno == errno.EXDEV:
            raise RuntimeError(
                f"refusing non-atomic copy: {tmp_name!s} -> {dest}"
            ) from exc
        raise

    # Best-effort directory fsync so the rename itself is durable.
    dir_fd = os.open(dest.parent, os.O_RDONLY)
    try:
        os.fsync(dir_fd)
    finally:
        os.close(dir_fd)


def assert_writable_same_device(temp_candidate: Path, dest: Path) -> None:
    if not same_device(temp_candidate, dest.parent):
        raise AssertionError(
            f"{temp_candidate} is device {os.stat(temp_candidate).st_dev}, "
            f"{dest.parent} is device {os.stat(dest.parent).st_dev}"
        )

A repro that should fail on a tmpfs /tmp and pass with the helper above:

# labeled demonstration: the broken helper vs the same-dir helper
import shutil
import tempfile
from pathlib import Path

DEST = Path("/tmp/demo-config-dir/config.json")  # change to a path on another mount
DEST.parent.mkdir(parents=True, exist_ok=True)

def broken_write(dest: Path, text: str) -> None:
    with tempfile.NamedTemporaryFile("w", delete=False, encoding="utf-8") as handle:
        handle.write(text)
        handle.flush()
        os.fsync(handle.fileno())
        tmp = handle.name
    shutil.move(tmp, dest)  # copies across devices; readers can observe a partial file

# Watch a reader in another terminal:
#   python -c "import time,pathlib; p=pathlib.Path('/path/config.json')\nwhile True:\n    print(repr(p.read_text()[:40])); time.sleep(0.01)"

If you want a mechanical check before the writer even starts, compare devices and refuse to boot.

python - <<'PY'
import os, tempfile, sys
src = tempfile.gettempdir()
dst = "/var/app"
if os.stat(src).st_dev != os.stat(dst).st_dev:
    print("default tempfile dir is not on the dest filesystem")
    sys.exit(2)
print("same device; default tempfile dir is acceptable")
PY

Decision table I wish I had on hour one

Situation	What happens	Use this
Tempfile and dest share st_dev	os.replace / rename is atomic for the name	Same-dir tempfile, then os.replace
/tmp is tmpfs, dest is a disk	os.replace raises EXDEV; shutil.move copies	mkstemp(dir=dest.parent)
You catch EXDEV and copy	Readers can see a prefix of the new file	Do not catch it unless you lock readers
You need durability across power loss	rename atomicity is not the same as fsync	fsync file, then fsync directory
Dest is object storage or NFS with weird rename	POSIX assumptions leak	A lock, a generation file, or a real store

Numbered test plan I would actually keep in the repo:

1. Create two directories you know are on different devices, even if one is a bind mount over tmpfs.
2. Write with the broken shutil.move helper while a reader loops on read_text().
3. Confirm the reader can print a truncated prefix at least once.
4. Switch to mkstemp(dir=dest.parent) plus os.replace, and confirm strace shows rename / renameat.
5. Confirm the helper raises if someone points the tempfile directory back at /tmp.
6. Repeat on a machine whose findmnt -T /tmp does not match findmnt -T /var.

Bind-mount trick when you only have one disk, which I needed on the laptop:

sudo mkdir -p /mnt/tmpfs-tmp /var/app
sudo mount -t tmpfs -o size=32M tmpfs /mnt/tmpfs-tmp
# point TMPDIR at the tmpfs and keep the dest on the disk
export TMPDIR=/mnt/tmpfs-tmp
stat -c '%d %n' "$TMPDIR" /var/app

What broke in the first helper

NamedTemporaryFile defaults to tempfile.gettempdir(), and that path is a policy, not a promise. delete=True also unlinks the name on POSIX while the descriptor stays open, which is a second footgun if a child process receives the path. shutil.move looks atomic in a code review because the name contains "move", and reviewers rarely ask where /tmp is mounted. I made every one of those mistakes in the same helper, then spent a day blaming NFS for a copy I had requested.

Would I still fsync? Yes, because atomic rename does not flush dirty pages for you. Would I still log st_dev on startup? Yes, because that one line would have ended hour zero.

What I would repeat

• Print st_dev for the tempfile directory and the destination during process boot, every time.
• Put tempfiles next to the live file, with a dotted prefix, so a crash leaves obvious junk.
• Refuse EXDEV instead of copying, unless the reader protocol can tolerate a partial object.
• Keep a two-line strace recipe in the runbook, because rename versus copy is visible in seconds.
• Reproduce on a box whose mount table is uglier than a developer laptop, before calling the bug unreproducible.

Limitations, and who should not copy this

This note is POSIX-shaped. Windows replace semantics, sharing modes, and tempfile defaults are a different argument, and I did not verify them here. Directory fsync is best-effort and not meaningful on every filesystem, so do not treat it as a durability certificate. If your destination is S3, NFS with broken rename, or a FUSE mount with interesting caching, same-directory os.replace will not invent a consistency model you do not have.

Do not use this approach if you need a locked multi-writer protocol, because rename does not queue writers for you. Do not dump production configs, tokens, or customer files onto a shared free server just to inspect st_dev. A local tmpfs bind mount is enough to prove the copy path, and it keeps secrets on your own disk.

I still trust os.replace. I just stopped trusting it to teleport a file across a mount point and remain atomic. If your laptop cannot reproduce a torn file, ask findmnt what your laptop is hiding, and only then start blaming NFS.