I Watched os.replace Fail for 48 Hours. The Tempfile Lived on Another Device

Have you ever watched a supposedly atomic write pass every local check and then fail the moment it left your laptop? I just burned forty-eight hours on that mismatch, and the traceback looked almost insultingly small. The helper created a tempfile, wrote JSON bytes, and called os.replace, which every review comment still called atomic. On my laptop the rename stayed on one filesystem, but the remote workspace did not share that luck.

This note is a field log, not a victory lap. I wrote down what I tried, what actually broke, and the tiny check I will refuse to skip next time. If you ship config files, lock files, or any replace-in-place JSON, you already own this class of bug. The interesting part is how quietly it hides until st_dev disagrees.

Hour 0–8: the helper that looked finished

I needed a config writer that would not leave a truncated JSON file if the process died mid-write. That sounds boring until a service reads the file on boot and treats truncated JSON as a fatal parse error. I asked a coding assistant for an atomic replace helper, then I pasted the result into a tiny module without questioning the tempfile directory. The generated pattern looked like every snippet I had already seen on the internet, so I trusted it.

Does that sound familiar, or am I the only person who reviews the write loop and ignores tempfile.gettempdir()? Local tests wrote a file, read it back, and printed a smug ok. I even killed the process with SIGKILL during the write and still got either the old file or the new file. Nothing in that first evening suggested a second filesystem was waiting offstage.

Here is the helper I actually shipped into the first remote run. I am labeling it as the broken draft, because this is the version that passed on one disk and exploded on two.

# broken_atomic.py — reconstructed from the first draft I ran
from __future__ import annotations

import json
import os
import tempfile
from pathlib import Path
from typing import Any


def atomic_write_json(path: str | Path, payload: Any) -> None:
    path = Path(path)
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, tmp_name = tempfile.mkstemp(suffix=".json")
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as handle:
            json.dump(payload, handle, indent=2, sort_keys=True)
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(tmp_name, path)  # looks atomic. is not, across devices.
    except Exception:
        try:
            os.unlink(tmp_name)
        except OSError:
            pass
        raise

I ran it locally with a throwaway target under my home directory, which is the kind of path reviewers forget to interrogate. Why would I interrogate /tmp when the test output already said the file existed?

python3 - <<'PY'
from broken_atomic import atomic_write_json
atomic_write_json("atomic-demo.json", {"retry": 3, "ok": True})
print(open("atomic-demo.json", encoding="utf-8").read())
PY

Hour 8–24: the remote run that refused the rename

The next morning I wanted a second machine, not another unit test on the same disk. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access to draft the helper, then I reran the same script on the free server option so I was not debugging against one laptop filesystem. I did not need a named model, a quota slide, or a hardware tour for that step. I needed a different st_dev.

The remote traceback was OSError: [Errno 18] Invalid cross-device link. Have you stared at EXDEV long enough to distrust every blog post that treats os.replace as a complete atomic protocol? I had not. I blamed permissions, then SELinux, then a stale .pyc, then the JSON encoder, then the fact that the target path lived under a workspace directory I had not created by hand.

None of those theories survived a two-line probe. The tempfile landed in the default temp directory, and the destination lived on another mount. os.replace can overwrite on the same filesystem, but it cannot teleport an inode across devices. POSIX has been saying that out loud for decades, and I still had to hear it from errno.

python3 - <<'PY'
import os, tempfile
from pathlib import Path

tmp = tempfile.gettempdir()
target_dir = Path.cwd()
print("TMPDIR / gettempdir:", tmp)
print("cwd:", target_dir)
print("tmp  st_dev:", os.stat(tmp).st_dev)
print("cwd  st_dev:", os.stat(target_dir).st_dev)
print("same device:", os.stat(tmp).st_dev == os.stat(target_dir).st_dev)
PY

df -P "$(python3 -c 'import tempfile; print(tempfile.gettempdir())')" "$PWD"
stat -c '%d %n' "$(python3 -c 'import tempfile; print(tempfile.gettempdir())')" "$PWD"

When those two device IDs differ, the rename is not a rename anymore. It is a polite refusal. My laptop had been lying by accident, because /tmp and $HOME shared a device there. The remote session did not extend that courtesy.

Hour 24–36: the false fixes I will not repeat

I tried Path.rename first, because the pathlib docs read like a cleaner os.rename. Same EXDEV. I tried os.rename instead of os.replace, as if the older name had a secret cross-device tunnel. Same EXDEV. I tried shutil.move, which did “work,” and that is the most dangerous kind of work.

shutil.move copies across devices and then unlinks the source. You get a file at the destination, yes, but you lose the atomic swap on the destination filesystem. A crash in the copy window can still leave a truncated object, which was the whole reason I wanted os.replace. So did I fix atomicity, or did I just hide the errno behind extra syscalls?

I also tried setting TMPDIR to the workspace, which helped one process and broke another tool that assumed /tmp was shared scratch. Environment variables are not an atomic-write protocol. They are a footgun with a helpful name.

What actually broke, in one list I wish I had written at hour eight:

• tempfile.mkstemp() without dir= put the inode on the default temp mount.
• os.replace(src, dst) requires src and dst to live on the same device.
• Local green tests never compared os.stat(...).st_dev, so they could not fail.
• shutil.move restored availability and deleted the only signal I needed.
• Killing the process locally still looked atomic, because both paths shared a device.

Hour 36–48: the artifact I will keep

The fix is not clever. Create the temporary file on the destination directory's filesystem, fsync the bytes, then os.replace within that directory. Same device, same directory, one inode swap. I want the tempfile to be a sibling of the target, not a tourist from tmpfs.

# atomic_write.py — sibling tempfile, then replace
from __future__ import annotations

import json
import os
import tempfile
from pathlib import Path
from typing import Any


def atomic_write_json(path: str | Path, payload: Any) -> None:
    path = Path(path).resolve()
    path.parent.mkdir(parents=True, exist_ok=True)

    fd, tmp_name = tempfile.mkstemp(
        suffix=".json.tmp",
        prefix=f".{path.name}.",
        dir=path.parent,  # stay on the destination device
    )
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as handle:
            json.dump(payload, handle, indent=2, sort_keys=True)
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(tmp_name, path)
    except Exception:
        try:
            os.unlink(tmp_name)
        except OSError:
            pass
        raise


def assert_same_device(tmp_name: str, dest: Path) -> None:
    if os.stat(tmp_name).st_dev != os.stat(dest.parent).st_dev:
        raise RuntimeError(
            f"temp {tmp_name!r} and dest {dest!r} are on different devices"
        )

The assert_same_device helper is optional in production, and I still want it in tests. If a future refactor “simplifies” the tempfile directory, the test should scream before a remote boot does. Would you rather fail in CI, or fail while a service is parsing half a config?

Reproducible test plan

I now run this plan on any machine that claims the helper is safe. It is boring on purpose, because boring is the point of an atomic write.

1. Print tempfile.gettempdir(), Path.cwd(), and both st_dev values before any write.
2. Write with the sibling-temp helper and confirm the target parses as JSON.
3. Force a cross-device case if the platform allows it: point a broken helper at a path whose st_dev differs from gettempdir(), and expect EXDEV.
4. Interrupt during the write with SIGKILL and confirm the target is either absent, intact old JSON, or intact new JSON.
5. Refuse shutil.move as a green result unless you explicitly accept copy semantics.

# test_atomic_device.py — run with: python3 test_atomic_device.py
from __future__ import annotations

import json
import os
import tempfile
from pathlib import Path

from atomic_write import atomic_write_json


def main() -> None:
    workspace = Path.cwd() / "_atomic_workspace"
    workspace.mkdir(exist_ok=True)
    target = workspace / "config.json"
    atomic_write_json(target, {"host": "local", "n": 1})
    loaded = json.loads(target.read_text(encoding="utf-8"))
    assert loaded["n"] == 1

    tmp_dev = os.stat(tempfile.gettempdir()).st_dev
    dst_dev = os.stat(workspace).st_dev
    print("temp device", tmp_dev, "dest device", dst_dev)
    print("cross-device tempfile would EXDEV:", tmp_dev != dst_dev)
    print("ok")


if __name__ == "__main__":
    main()

Decision table I wish I had taped to the monitor

Move method	Same device	Different device	Crash window	Use when
os.replace(tmp, dest) with tmp in dest.parent	Atomic replace	Should never happen	Tiny, after fsync	Default for local files
os.replace(tmp, dest) with tmp in /tmp	Atomic replace	Raises EXDEV	None, it fails closed	Only if you proved st_dev matches
shutil.move	Rename	Copy plus unlink	Copy can truncate dest	Last resort, not “atomic”
Write directly to dest	In-place	In-place	Readers see partial JSON	Never for boot-critical config

What I would repeat, and what I would not

I would repeat the st_dev print before I trust any tempfile plus replace combo. I would repeat putting dir=path.parent into mkstemp, even when it makes the workspace look slightly dirtier. I would repeat fsync before replace, because a rename of unflushed buffers is a different lie. I would repeat a remote run that is not the same disk as my editor, because one laptop is a single filesystem anecdote.

I would not repeat treating assistant output as a storage tutorial. The model can emit a plausible helper in seconds, and it cannot see your mount table. I would not repeat using shutil.move as a quiet fallback that paints over EXDEV. I would not repeat green tests that never compare devices. If the test cannot fail, it is documentation, not coverage.

Limitations, and who should not copy this

This sibling-temp plus os.replace pattern is for POSIX-style local filesystems where rename in one directory is atomic. It does not make object storage atomic, and it does not make a multi-document transaction. NFS, FUSE, and some networked mounts have rename semantics that are weirder than EXDEV, so I still want an application-level checksum if readers cannot tolerate a stale file. Windows adds sharing rules on top of volume boundaries, and this note does not pretend to be a Win32 locking guide.

Do not use this approach if your destination is S3, GCS, or another HTTP blob store. Do not use it if multiple writers must coordinate beyond last-replace-wins. Do not use it as a substitute for a real database when the file is a concurrent queue. And do not use it if you cannot fsync, because then you are only atomic in memory.

The forty-eight hours were not about JSON. They were about a tempfile that commuted in from another device while I kept reviewing the encoder. Next time the helper looks finished, I am printing two integers first. If those st_dev values disagree, os.replace is not a strategy. It is a future traceback.