DEV Community: Qian_Xiao

Why Disk Space Can Stay Full After a Huge Log File Is Deleted

Qian_Xiao — Mon, 26 Oct 2026 20:48:41 +0000

A server runs out of disk space. The largest log file is found, deleted, and confirmed absent from its directory. Yet df still reports the filesystem as full.

The common misconception is simple: deleting a file immediately releases all of its disk space.

On Linux, deletion usually removes a directory entry—a name that points to a file’s inode. The file’s stored data can be reclaimed only when two conditions are satisfied:

No directory entries refer to the inode.
No process still has the file open.

If a running application opened a log file before it was deleted, its file descriptor remains connected to the same inode. The pathname may have disappeared, but the underlying file still exists for that process. The application can continue writing to it, and the supposedly deleted file may even keep growing.

This behavior is intentional. Open file descriptors remain valid when files are renamed or unlinked, allowing programs to use files safely without depending on their names remaining unchanged.

The mismatch between common disk tools makes the problem especially confusing. du walks the directory tree and totals files it can reach by pathname. An unlinked file has no pathname, so du cannot see it. df, by contrast, reports allocated filesystem blocks. Those blocks are still allocated while a process holds the file open. As a result, du may account for far less data than df says is in use.

The usual diagnostic command is:

lsof +L1

This lists open files whose link count is below one, including files marked (deleted). The process name, PID, file descriptor, and file size help identify which application is retaining the space.

The safest remedy is generally to make that application close and reopen the file. Depending on the program, that may mean requesting a log reopen, performing a graceful reload, restarting a service, or restarting a container. Once the final open file descriptor is closed, the kernel can reclaim the blocks.

Killing a process is not always necessary. Many daemons support log rotation explicitly: the old log is renamed, a signal or reload command asks the daemon to reopen its log path, and the rotated file is then compressed or removed. Without the reopen step, log rotation can leave the daemon writing through its old descriptor.

There is also an emergency technique: truncating the open descriptor through /proc/<pid>/fd/<fd>. That can release space without terminating the process, but it should be used cautiously. Writing to another process’s file descriptor can interact badly with buffered output, file offsets, or application assumptions.

The key distinction is between a filename and an open file. rm removes the filename. Storage is reclaimed only after the filesystem has neither a remaining name nor a process reference to the file.

Why Linux Load Average Can Rise While CPUs Stay Mostly Idle

Qian_Xiao — Wed, 21 Oct 2026 16:04:33 +0000

A high Linux load average does not necessarily mean the processors are busy. The common claim that “load average measures CPU usage” confuses two related but distinct signals.

Linux calculates load average from the number of tasks that are runnable and the number stuck in uninterruptible sleep. Runnable tasks are executing on a CPU or waiting for one. Tasks in uninterruptible sleep, usually shown as state D by tools such as ps, are waiting for a kernel operation to finish and cannot currently handle ordinary signals.

Storage I/O is a frequent cause of uninterruptible sleep. Suppose many processes read from a slow or unavailable disk. Those processes may spend most of their time waiting rather than executing instructions. CPU utilization can remain low, yet the load average rises because Linux still includes those waiting tasks in its load calculation.

This distinction explains why load average and CPU utilization must be read separately. CPU utilization reports how much processor time is being consumed. Load average reflects sustained demand from runnable tasks plus certain blocked tasks. It is therefore possible to observe a load average of 20 alongside mostly idle CPUs when many processes are waiting in D state.

The three familiar values in /proc/loadavg and commands such as uptime are exponentially weighted averages over approximately one, five, and fifteen minutes. They are not instantaneous counts. A short disturbance can keep the one-minute value elevated after the underlying problem has disappeared, while the longer values reveal whether the pressure is persistent.

CPU count still matters when runnable work is responsible for the load. A load near eight may indicate saturation on an eight-CPU system because roughly eight tasks can run simultaneously. But that interpretation fails if much of the count comes from uninterruptible sleepers. Dividing load average by CPU count does not transform it into a reliable utilization percentage.

When load is high but CPU usage is low, inspect task states before adding processor capacity. Commands such as ps can reveal D-state processes, while storage latency, network-backed filesystems, device failures, and kernel waits may identify the resource actually causing the queue.

Load average is best treated as a pressure clue. It says that tasks are accumulating, but CPU metrics and process states are needed to determine why.

Docker Stop Signals Reach PID 1, Not Every Process in the Container

Qian_Xiao — Fri, 16 Oct 2026 16:57:48 +0000

A common misconception says that docker stop sends SIGTERM to every process inside a container. It does not. The signal is sent to the container’s main process: PID 1 in the container’s PID namespace.

This distinction matters when PID 1 is not the application that needs to shut down gracefully. Consider an image with this instruction:

ENTRYPOINT service --listen

This is Dockerfile shell form. Docker normally starts it through a shell equivalent to:

/bin/sh -c 'service --listen'

The shell becomes PID 1, while service runs as its child. When the container is stopped, the runtime signals the shell. Whether the application receives that signal now depends on the shell’s behavior. A non-interactive shell may not forward SIGTERM to its child. The service can therefore miss its opportunity to stop accepting work, flush buffers, close connections, or persist state.

After the configured grace period expires, Docker sends SIGKILL to the container. SIGKILL cannot be caught or handled, so the application is terminated immediately.

Exec form removes this unnecessary intermediary:

ENTRYPOINT ["service", "--listen"]

Here, service itself becomes PID 1 and directly receives the stop signal. A wrapper script can also work, but it should replace itself with the application by using exec:

#!/bin/sh
prepare_environment
exec service --listen

Without exec, the wrapper remains PID 1 and must explicitly forward signals.

PID 1 also has a special process-management responsibility. Orphaned descendants are reparented to it, and it must reap exited children to prevent zombie processes. Applications that spawn subprocesses but do not handle this role well may benefit from a small init process such as Docker’s --init option. That init process can forward signals and reap children while remaining PID 1.

The precise rule is simple: stopping a container signals its designated main process, not every process in the container. Graceful shutdown depends on ensuring that PID 1 either is the application or correctly forwards signals to it.

A Failed Kubernetes Readiness Probe Does Not Restart the Container

Qian_Xiao — Sun, 11 Oct 2026 20:58:27 +0000

A common misconception says that when a Kubernetes readiness probe fails, Kubernetes restarts the container. It does not. Readiness controls whether a Pod should receive traffic from Services; container restarts are the job of liveness probes and process supervision.

The kubelet runs a container’s readiness probe according to the probe configuration. If the probe succeeds, Kubernetes can mark the container as ready. If every required readiness condition is satisfied, the Pod’s Ready condition becomes true. When the probe begins failing, the container is marked not ready, which normally makes the Pod not ready as well.

That state change affects service discovery. Controllers update the relevant EndpointSlices so that a not-ready Pod is no longer treated as an eligible backend for ordinary Service traffic. Components such as kube-proxy, or another implementation of Service routing, then converge on the new endpoint state.

The container keeps running throughout this process. Its process is not killed, its filesystem is not reset, and its restart count does not increase merely because readiness failed. The kubelet continues executing the readiness probe. If the application recovers and the probe succeeds again, the Pod can become ready and rejoin the Service’s available backends without a restart.

This distinction makes readiness probes suitable for temporary inability to serve traffic. An application may be alive but unable to answer requests while it refreshes configuration, reconnects to a dependency, fills a cache, or drains work. A readiness failure can isolate that instance while preserving its in-memory recovery process.

Removal from Service endpoints is not an instantaneous cancellation mechanism. Endpoint updates must propagate, and existing connections may remain open. A client using a persistent TCP connection might continue sending requests to the same Pod even after that Pod becomes unready. Readiness is therefore a routing signal, not a guarantee that all traffic stops at the exact moment a probe fails.

If an unhealthy container must be terminated and started again, a liveness probe expresses that policy. A liveness failure can cause the kubelet to restart the container according to the Pod’s restart policy. Using liveness for a condition that can recover naturally may instead create restart loops and erase useful recovery progress.

The precise correction is simple: readiness failure removes a Pod from normal Service traffic; it does not restart the container.

A Bind Mount Hides Existing Container Files; It Does Not Merge with Them

Qian_Xiao — Fri, 09 Oct 2026 19:43:21 +0000

A common misconception about container bind mounts is that mounting a host directory over a container directory combines the contents of both locations. It does not. The mount replaces the directory’s visible view for as long as the mount exists.

Suppose an image contains these files:

/app/config/default.yaml
/app/config/schema.json

The container is then started with a host directory mounted at /app/config:

docker run --mount type=bind,src=/srv/config,dst=/app/config example

If /srv/config contains only production.yaml, the container sees:

/app/config/production.yaml

It does not also see default.yaml or schema.json. Those image files have not been deleted, copied, or overwritten. They are merely obscured by the mount.

This behavior comes from Linux mount semantics rather than a container-specific merge algorithm. A mount attaches one filesystem tree at a mount point. Path lookup beneath that point proceeds through the mounted tree, so the directory entries from the underlying image layer are no longer reachable through their original paths.

That distinction explains a frequent failure mode: an application works when launched directly from its image, then fails after a configuration directory is mounted. The operator expected to override one file, but mounting the directory hid every image-provided file under that path.

A bind mount also differs from the image’s layered filesystem. Image layers can contribute files to a combined container root filesystem, but a bind mount is applied afterward at a particular path. It does not become another union layer whose entries are merged with the directory underneath.

If only one file should be replaced, mount that file instead of its parent directory:

docker run \
  --mount type=bind,src=/srv/config/production.yaml,dst=/app/config/default.yaml,readonly \
  example

Another option is to place defaults outside the mount point and have an entrypoint copy them into a writable directory before starting the application. The exact design depends on whether configuration should be immutable, generated, or persistent.

Stopping and removing the container removes the mount relationship, not the underlying image files. A new container started without that bind mount sees the original files again.

The corrected claim is precise: a bind mount does not merge host and container directory contents. It makes the mounted source the visible filesystem tree at the destination, hiding whatever was previously visible there.

Was bash the wrong language for my agent?

Qian_Xiao — Thu, 17 Sep 2026 20:36:47 +0000

I have a small agent that handles one piece of routine work at a time. It looks
at what needs doing, picks the thing most worth doing, shows me the plan, and
does it if I say yes. Underneath, it is glue: it drives a few command-line
tools, calls a model, reshapes a lot of JSON, and prints a readable summary.

It was 2150 lines of bash across seven files. It is now Python.

So the answer looks like yes. I don't think it is, and why I don't is most of
the reason I'm writing this down.

The argument for staying was sound. Its inputs weren't.

I had been through this question before and decided to stay — carefully enough
that the reasoning became a section of the project README titled Why this is
still bash. Nine tenths of the program is subprocess orchestration, which is
bash's home ground. Rewriting 2000 lines with no test coverage is the standard
way to lose behavior silently. And the bug ledger said the expensive bugs were
design errors that any language would have permitted.

I still think all of that is true.

What moved was a requirement. I had been treating runs with no build step as
hard, which made "python3 is already installed" the load-bearing argument. Then
the requirement got clarified: no build step isn't a rule, it just shouldn't be
complicated to start.

That one sentence killed my best argument. So I measured what was actually at
stake — 89ms for Python plus every standard library module it needs, against
3ms for bash. Eighty-six milliseconds, in a program that waits thirty to a
hundred seconds on a model.

My reasoning was valid; its inputs weren't, and I had spent almost no effort
checking them. That ratio was backwards, and I don't think that's unusual.

What decided it was 162 calls to `jq`

Every list length, every filter, forking a process to handle data that should
have been sitting in memory.

The cost was not performance. 162 forks are nothing next to a minute of model
latency. The cost was expressiveness. Every structure in the program either
fit in a one-line jq expression or got split into three pieces. What I wrote
was never the structure I wanted; it was the structure jq could state on one
line. That cost is invisible in any single line of code and shows up in the
designs you never consider.

I had three lists: the files a change actually touched, the files the agent
itself had written, and the files I had approved in advance. I needed two
differences between them.

later="$(jq -nc --argjson a "$actual" --argjson w "$written" \
    'if $w == null then [] else ($a - $w) end')"
extra="$(jq -nr --argjson w "$written" --argjson p "$planned" \
    '($w - $p) | join(", ")')"

later = [] if written is None else [f for f in actual if f not in written]
extra = [f for f in written if f not in planned]

The second one is barely shorter. It is what I would have written on the first
attempt; the first took me several tries to get right.

There was also a run that died on line 567: 1: command not found, which I
never located. It went away when that section was rewritten for unrelated
reasons. A bug you can't find after the fact doesn't just go unfixed — it
tells you the next one of its kind will too.

The line count did not go down

The port took a day.

bash      2150 lines
Python    2258 lines   ← up 108

            of which:
  code      1278       ← down 40%
  comments   980

Most people expect the opposite, so it's worth being blunt about. The code
shrank by forty percent; the difference is comments and docstrings — the notes
recording which specific incident each safety check exists to prevent, which
were exactly what I'd been afraid of losing.

Evaluate this rewrite by total line count and it accomplished nothing.

What made it cheap: two seams built for other reasons

This is the part I actually wanted to write down.

The prompts are files, not strings. Every prompt lives in its own Markdown
file and the code fills {{placeholder}} holes in it. I did that for unrelated
reasons: prompts get edited constantly, they want to be read as prose, and a
stray $ or backtick has to stay inert instead of being eaten by the shell.

The result was that the migration did not touch one word of any prompt. I
checked modification times afterward to be sure. Everything that determines
this program's behavior — the criteria I've tuned over and over, the order
judgments get made in — lives in those files. Changing languages only replaced
the glue that assembles them.

Each kind of task is a separate executable that speaks JSON. Verb on argv,
JSON on stdin, JSON on stdout. I built it that way because bash has no modules,
and putting them behind a process boundary beat having them scribble on each
other's variables. Pure coping.

The result: there was no big-bang rewrite to choose. The orchestrator could
be Python while the task types were still bash, or the reverse. I moved one
file at a time and ran each one on its own afterward. (That boundary is
probably also why this never hit the wall bash projects hit — the largest bash
agent I know of reached 4700 lines as a single file assembled by cat src/*.sh,
and what broke was module structure, not correctness.)

Neither seam was built with portability in mind. Both were built to solve
something annoying at the time.

Whether a rewrite will be cheap is decided before you start it.

I kept the process boundary afterward, by the way. Folding the task types into
Python imports would save a JSON round trip, and it is the best structural
decision in the project.

The real risk of Python is not the build step

With a shebang and standard library only, it is invoked exactly the way it was
before. No virtualenv, no install.

The risk is that Python invites dependencies, and bash's poverty was itself
a form of protection. requests when urllib is right there; a schema
validator when the model CLI already enforces the schema; an argument parser
for 25 lines of parsing; a formatting library for a display layer that exists.
Each has a plausible case, and after all four "quick to start" is gone.

So there is one rule in the README now: standard library only, and say why it
can't be done with it before adding anything. The whole program needs six
modules.

What I did not verify

I exercised every path after the port, including a full dry run of the
expensive one — isolated checkout, model writes the code, formatter, vet,
build, tests, commit — stopping short of pushing. Fifteen tests on the pure
functions pass.

The two lines that push a branch and open a change request never ran,
because running them means actually opening one. And the safety checks inside
the task types I translated by hand, one at a time. I believe I got them right,
and this project still has no test that can prove it.

I don't think bash was the wrong choice. It carried this to 2150 lines, and for
all of that time I was changing judgment logic rather than fighting the
language. Its problem was never that it couldn't do the job. Its problem was
that its expressiveness had started deciding my designs.

Everything was running. The port belonged to the wrong process.

Qian_Xiao — Fri, 11 Sep 2026 18:24:59 +0000

My browser sat there spinning on "connecting".

The setup is common enough: x11vnc shares a screen on port 5900, and
websockify forwards to it so a browser can connect. I checked the services.
websockify was running. Everything was running. Nothing worked.

The question I actually needed answered was simple:

Who is using port 5900?

That's normally ss -ltnp or lsof -i :5900, and neither was installed on that
box — which is why I'd written a small tool for this one question:

$ portclue 5900
NOT EXPOSED LOCALLY

TCP port 5900

  127.0.0.1:5900/tcp  [NOT_EXPOSED_LOCALLY]
    -> LISTEN             ... bound to 127.0.0.1:5900/tcp
    -> OWNED              PID 1087636 (x11vnc), systemd unit session-1911.scope
    -> LOOPBACK_ONLY      127.0.0.1 is reachable only from this network namespace

An x11vnc had the port — just not the one I'd started. It was a leftover from
a session days earlier that had never shut down, so the new one could never get
the port, and websockify had been faithfully forwarding to a dead screen the
whole time.

(PortClue gave me the PID. ps is what confirmed the process was far older than
everything around it — the tool doesn't report process age yet.)

Why I reach for it

Same facts ss would give you, but written out instead of encoded.

127.0.0.1 isn't a number you have to interpret; it says "reachable only from
this network namespace". On a port bound to 0.0.0.0 it says ALL_INTERFACES,
then reads your nftables or iptables rules to see whether anything is actually
allowed through. If it can't read them, it says UNKNOWN instead of guessing.

It gets all of that without ss or lsof installed, by asking the kernel
directly.

It's read-only: it never connects to the port you ask about, and it can't kill
anything. Scope is Linux TCP listeners — that's the whole promise.

Try it

curl -fsSL https://raw.githubusercontent.com/pbxqdown/portclue/v0.1.2/scripts/install.sh | sh

portclue        # everything listening
portclue 5900   # one port, explained

https://github.com/pbxqdown/portclue