Your sed -i Edit Isn't Reaching the Container: The Bind-Mount Inode Trap

This article was originally published on Jo4 Blog.

We tweaked a scrape config in our Prometheus YAML, ran sed -i on the host, confirmed the file on disk had the new content, and waited for Prometheus to pick it up. It didn't. An hour later we were staring at a stat output that explained everything we'd ever misunderstood about bind mounts. If you've ever edited a file Docker mounted into a container and watched the container act like nothing happened, this post is for you.

The Symptom

Our Prometheus container has its config bind-mounted from the host:

services:
  prometheus:
    image: prom/prometheus
    volumes:
      - /opt/impress/prometheus.yml:/etc/prometheus/prometheus.yml

We needed to add a scrape target. Easy:

sed -i 's/scrape_interval: 15s/scrape_interval: 30s/' /opt/impress/prometheus.yml
docker exec prometheus kill -HUP 1   # signal Prometheus to reload

The host file looked correct. Prometheus's /-/reload endpoint returned 200. But the runtime config in the UI still showed 15s. We re-ran the SIGHUP. Same. We checked Prometheus logs — it said it had reloaded the config. From its own file. Which apparently still said 15s.

That's the moment we stopped trusting our assumptions and asked the container what it was actually seeing.

Diagnosis: stat Tells the Truth

We compared the two views of "the same file":

# On the host
$ md5sum /opt/impress/prometheus.yml
a3f9c1e8...  /opt/impress/prometheus.yml

$ stat /opt/impress/prometheus.yml
  File: /opt/impress/prometheus.yml
  Size: 2847       Blocks: 8          IO Block: 4096   regular file
Device: fc01h/64513d    Inode: 1835421     Links: 1

# Inside the container
$ docker exec prometheus md5sum /etc/prometheus/prometheus.yml
7b2d4f06...  /etc/prometheus/prometheus.yml

$ docker exec prometheus stat /etc/prometheus/prometheus.yml
  File: /etc/prometheus/prometheus.yml
  Size: 2791       Blocks: 8          IO Block: 4096   regular file
Device: fc01h/64513d    Inode: 1835098     Links: 0

Two things jumped out:

1. Different MD5s. The container is reading different bytes than the host file contains.
2. Links: 0 inside the container. Zero hard links means no directory entry points at this inode anywhere on the filesystem. The file has been unlinked. The container is the only thing keeping the inode alive.

Different inode numbers (1835421 on the host, 1835098 in the container) confirmed it: these aren't the same file anymore. They were, when the container started. They aren't now. The container is holding open a ghost — an inode that exists only because it still has an open file descriptor pointing at it.

Why sed -i Is the Culprit

This is the part of sed -i nobody reads the man page about. sed -i does not edit in place. Despite the flag name, it does this:

1. Open the target file for reading.
2. Open a temp file in the same directory (sedXYZ or similar).
3. Stream the transformed content to the temp file.
4. rename(2) the temp file over the target name.

That rename(2) is the killer. The original inode (the one with the old content) is no longer reachable by name, but it doesn't get freed because the container has it open. The directory entry /opt/impress/prometheus.yml now points at a brand new inode with the new content. On the host that's the same path with new bytes — looks fine. Inside the container, nothing changed: the bind mount was set up at container start against the original inode, and bind mounts of single files are bound to inodes, not directory entries.

So the container keeps reading the ghost. Forever. Or until you restart it.

This is also why Links: 0. The original directory entry is gone (it now points to the new inode), so the old inode has zero references in the directory tree — but it has one open file descriptor inside the container, so the kernel keeps it alive.

The Fix

There are two correct ways out.

Option A: edit in place for real

You want to write into the same inode, not replace it. cat > file does exactly that — it opens the existing inode with O_TRUNC and writes new content. The inode number doesn't change. The bind mount still points at it. The container sees the update immediately:

# Build new content somewhere, then truncate-and-write the SAME inode
sed 's/scrape_interval: 15s/scrape_interval: 30s/' /opt/impress/prometheus.yml > /tmp/prom.new
cat /tmp/prom.new > /opt/impress/prometheus.yml
rm /tmp/prom.new

Confirm with stat that the inode number didn't change. If it did, you used a tool that does atomic-rename. Back up and try again.

Option B: restart the container

If the file change is paired with anything else (image bump, env-var change, a sibling config edit), just rebuild the container. This is what we did:

docker compose up -d --force-recreate prometheus

--force-recreate gives us a fully fresh container — new bind mounts against current inodes, plus any env-var or compose-file changes get picked up. Less surgical than cat > file, but bulletproof when you're not sure what else drifted.

docker compose restart prometheus also works for the inode problem alone, but it won't pick up env-var or compose-file changes — it restarts the existing container in place.

Tools That Have the Same Trap

sed -i is the famous one, but the trap is everywhere atomic-rename is the safe-write idiom:

• mv newfile oldfile — literally rename(2). Same trap.
• vim and emacs by default — both write to a backup/swap file and rename over the target, for crash safety. Configurable, but the defaults bite.
• perl -i -pe '...' — same temp-file-and-rename dance as sed -i.
• awk '...' file > tmp && mv tmp file — the explicit version of the same thing.
• Most "safe save" code in editors and language tooling — gofmt -w, prettier --write, black, you name it. They rename for atomicity, which is correct for preventing half-written files on crash, and wrong for bind-mounted single files.

Anything that promises atomic writes is doing this. Atomic writes and bind-mounted single files are fundamentally incompatible.

The Bulletproof Workflow

After this incident we made two rules for editing bind-mounted config:

1. Prefer directory mounts over single-file mounts. A bind mount of a directory tracks the directory entries inside it, not specific inodes. Atomic-rename inside that directory works the way you expect. The cost is exposing sibling files to the container.
2. When a single-file mount is unavoidable, edit with cat > file. Build the new content in a tempfile somewhere outside the mounted path, then truncate-and-write the target. Verify with stat that the inode number is unchanged.

For our Prometheus case we kept the single-file mount (we don't want other files in /opt/impress visible to Prometheus) and added a tiny wrapper script that does the cat-redirect dance. No more sed -i on bind-mounted files. Ever.

Lessons Learned

• sed -i is a lie. It's not in-place. It's "write a temp file and rename." That rename(2) creates a new inode and orphans the old one.
• Bind mounts of single files are inode-bound. Whatever inode the directory entry pointed to at container-start time is the inode the container will hold forever. Replace that inode on the host and the container keeps the old one.
• Links: 0 on a file inside a container is the smoking gun. It means the container is the last reference to an inode that's been unlinked from the filesystem. You're looking at a ghost.
• md5sum on both sides of the mount is the fastest diagnostic. If host and container disagree, you've hit some flavor of this trap.
• Atomic-rename is correct for safety, wrong for bind mounts. Use cat > file for in-place truncate-and-write, or mount the parent directory instead of the single file.
• When in doubt, --force-recreate the container. It's the bigger hammer but it's reliable and it picks up env-var changes too.

---

Ever spent an hour chasing this? What was your tell? Drop it in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.