48-Hour Field Notes: os.replace Looked Atomic Until the Server Raised EXDEV

I spent forty-eight hours trusting os.replace because every notes page called it the atomic write. Have you ever shipped a supposedly safe state file that only failed after the disk layout changed under it? My laptop never complained, so I blamed permissions, umask, and a noisy container runtime before I blamed the kernel. The real error was sitting in errno the whole time: EXDEV, which means an invalid cross-device link.

The symptom I kept misreading

The writer was a small Python helper that dumped JSON to a tempfile, flushed the handle, and then called os.replace. On the laptop the helper looked boring, which is exactly how I want file I/O to look during a deploy. On the Linux box the same function raised OSError: [Errno 18] Invalid cross-device link and nothing else. Why did I refuse to read that number out loud during hour one of the mess?

I assumed replace could not fail if both paths existed and the process could write the destination directory. That assumption is the whole bug, and it hides inside a helper that looks completely ordinary in code review. Atomic rename is a single-filesystem operation, not a durability promise from the Python standard library itself. Different mount points mean different device IDs, and the kernel will not pretend those two numbers are the same.

What I tried in the first twenty-four hours

I chased ghosts in the usual order, and none of those ghosts were the device ID I needed.

1. I printed os.access on the destination directory and watched it return True like a taunt.
2. I compared umask values and file modes, because a mode-only failure is the story I already knew.
3. I reran the job as the same UID the service file used, still hoping the answer was EACCES.
4. I traced rename syscalls and still stared at renameat without asking which mounts owned the paths.

Does that list look painfully familiar to anyone who debuges production Python on a laptop first? It should, because it is the permission-shaped path we walk when the traceback mentions a file. Cross-device failures wear a permission costume until you print st_dev for both paths involved. I even restarted the container once, which of course changed nothing about tmpfs sitting on another device.

The device IDs that finally showed up

The laptop kept /tmp and the project directory on one disk, so rename never had to leave home. The Linux environment did not share that layout, and that single difference explained the entire forty-eight hour detour. /tmp was tmpfs, and the state directory lived on another mount, which is a completely ordinary server shape. os.replace then had to rename across devices, and Linux answered with EXDEV instead of a silent success.

Here is the check I now run before I trust any helper that advertises an atomic replace:

findmnt -o TARGET,SOURCE,FSTYPE /tmp "$(pwd)"
stat -c '%d %m %n' /tmp "$(pwd)"
df -hT /tmp "$(pwd)"
python3 - <<'PY'
import os, tempfile
print("tmp", tempfile.gettempdir(), os.stat(tempfile.gettempdir()).st_dev)
print("cwd", os.getcwd(), os.stat(".").st_dev)
PY

If those device numbers differ, os.replace between them is not atomic, and it is not even allowed. shutil.move will copy and unlink instead, which is a different failure mode during a crash in the middle. Have you checked whether your CI runner, your laptop, and your container actually share one st_dev for /tmp?

A local repro you can run

This is a labeled local repro, not a production benchmark and not a claim about any hosted fleet. Run it on any machine where /tmp and the working directory might disagree about their device identity.

# repro_exdev.py — local repro for cross-device os.replace
import json
import os
import tempfile
from pathlib import Path

DEST_DIR = Path("state")
DEST_DIR.mkdir(exist_ok=True)
dest = DEST_DIR / "config.json"


def naive_write(payload: dict) -> None:
    # Intentionally uses the process temp dir, often another filesystem.
    fd, tmp_name = tempfile.mkstemp(prefix="config-", suffix=".json")
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as fh:
            json.dump(payload, fh)
            fh.flush()
            os.fsync(fh.fileno())
        os.replace(tmp_name, dest)  # OSError EXDEV when st_dev differs
    except OSError as exc:
        print(f"naive_write failed: errno={exc.errno} {exc.strerror}")
        raise
    finally:
        if os.path.exists(tmp_name):
            os.unlink(tmp_name)


def same_dir_write(payload: dict) -> None:
    # Temp file lives next to the destination, so rename stays on one device.
    fd, tmp_name = tempfile.mkstemp(
        prefix="config-", suffix=".json", dir=DEST_DIR
    )
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as fh:
            json.dump(payload, fh)
            fh.flush()
            os.fsync(fh.fileno())
        os.replace(tmp_name, dest)
    finally:
        if os.path.exists(tmp_name):
            os.unlink(tmp_name)


if __name__ == "__main__":
    tmp_dev = os.stat(tempfile.gettempdir()).st_dev
    dst_dev = os.stat(DEST_DIR).st_dev
    print(f"tempdir={tempfile.gettempdir()!r} st_dev={tmp_dev}")
    print(f"dest   ={str(DEST_DIR.resolve())!r} st_dev={dst_dev}")
    print("same device" if tmp_dev == dst_dev else "CROSS-DEVICE: replace will fail")
    payload = {"ok": True, "source": "repro"}
    try:
        naive_write(payload)
        print("naive_write succeeded on this host")
    except OSError:
        print("naive_write raised, as expected on split filesystems")
    same_dir_write(payload)
    print(dest.read_text(encoding="utf-8"))

Want a cheap way to force the failure on a single-disk laptop that keeps lying to you? Bind a tmpfs over a staging directory and point TMPDIR at that mount before you run the script.

mkdir -p /tmp/exdev-scratch state
sudo mount -t tmpfs -o size=32M tmpfs /tmp/exdev-scratch
TMPDIR=/tmp/exdev-scratch python3 repro_exdev.py
strace -e rename,renameat,renameat2 TMPDIR=/tmp/exdev-scratch python3 repro_exdev.py
sudo umount /tmp/exdev-scratch

I did not need that tmpfs trick on the remote box, because default /tmp already lived elsewhere. The laptop kept succeeding because both paths shared one disk, which is the most expensive kind of green test. After the mount table is visible, the traceback stops looking mysterious and starts looking like POSIX.

The same-directory pattern I will actually keep

The durable rule is short, and I wish I had written it on a sticky note before the first deploy.

• Create the tempfile in the destination directory, not in tempfile.gettempdir() by accident.
• Write the payload, call flush(), then os.fsync the file descriptor before close.
• Call os.replace only after that fsync, so readers never observe a truncated JSON object.
• Do not treat shutil.move as a drop-in atomic fix, because the copy fallback is not atomic.
• Log st_dev for both paths whenever a rename fails, before you blame permissions or umask.

Should you fsync the directory too after the replace returns, especially on filesystems that reorder metadata? On some Unix layouts, yes, if you cannot afford to lose the directory entry after a power cut. For a small config file I fsync the file and accept the remaining crash window as an explicit choice. That is a tradeoff I can defend in a review, not a law I pretend the kernel documented for Python users.

Why shutil.move did not save me

I almost swapped os.replace for shutil.move and called the incident closed after a green laptop run. shutil.move catches EXDEV and copies, which means a crash can leave two files or a half-written destination. Readers can also observe the copy in progress unless you copy to a sibling tempfile and then replace inside the destination filesystem. If the whole point was atomicity, a silent copy fallback is not a fix; it is a different bug with a friendlier traceback.

Where a spare shell and a free model actually helped

I could not reproduce EXDEV on the laptop, which is the most expensive kind of works-here story. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free server option as a second Linux environment so I could compare stat output against my laptop without guessing at mount tables. The free model access was useful for walking the traceback and the rename behavior, not for inventing a filesystem I had refused to inspect.

The model did not magically know my mounts, and it should not be trusted as if it did. I still had to paste df -hT, the errno, and the two st_dev numbers before the explanation snapped into place. That pairing is the workflow I would repeat: a second machine that is allowed to differ, plus a model that is allowed to be wrong until stat agrees. If you need a spare Linux shell to compare those numbers against a laptop that will not fail, MonkeyCode's free server option is what I used while writing this down.

A small decision table I keep now

Situation	What I do	Atomic on POSIX?
Temp and dest share st_dev	os.replace after fsync	Yes, for the rename
Temp lives on tmpfs, dest does not	mkstemp(dir=dest_dir) then replace	Yes, after the fix
I need to move across mounts	Copy to a sibling temp, fsync, replace, unlink source	Not during the copy
Several processes write the same file	Same-dir replace plus an advisory lock	Replace is atomic; writers still need a protocol
Crash leftover *.tmp files matter	Startup glob cleanup plus same-dir replace	Cleanup is best-effort

That table is not a benchmark, and it is not a claim about throughput on anybody's hardware. It is the checklist I wish I had opened at hour three instead of restarting a container. Would I still keep the naive mkstemp() call in a scratch script? Yes, but never in the path that publishes a config file other processes will read.

Limitations, and who should skip this

This pattern is for small state files and config JSON, not for multi-gigabyte artifacts that would punish a same-directory tempfile. Same-directory tempfiles can dirty the destination volume with short-lived names, which some backup tools will notice and some antivirus scanners will lock. If your destination is an object store, os.replace is the wrong primitive, because there is no POSIX rename hiding behind that HTTP API.

Skip this approach when you already write through a database that has its own durability story and crash recovery. Skip it when the file is a log you append, because rotation races are a different class of bug than replace. Skip it if you cannot trust fsync, for example on some network filesystems that ignore the call or that turn locks into polite suggestions. And please skip blaming Python, Docker, or umask until you have printed both device IDs next to the failing paths.

I also would not use a remote coding environment as the only production replica, because a spare Linux shell is a different machine rather than your fleet. SELinux labels, user namespaces, and volume drivers can still disagree after st_dev matches, and those disagreements will not show up in a two-line stat helper. The repro above tells you whether rename can be atomic. It does not tell you whether the surrounding service manager will keep the file.

What I would repeat

I would print st_dev in the exception handler on the first OSError, not after a day of permission theater that never had a chance to help. I would keep tempfile creation in the destination directory as the default, even on a laptop whose single disk cannot fail the naive path. I would still borrow a second Linux shell when my laptop and the failing host disagree, because mount tables are not portable folklore you can memorize once.

Would I still read man pages beside a model session while the clock is running? Yes, with the same rule I use for any untrusted summary: verify against errno, stat, and a script I can rerun. The forty-eight hours were not a mystery about JSON encoding, container UIDs, or a cursed umask value. They were a mystery about two device numbers I refused to print, and about a laptop that kept passing a test the kernel never owed me.