Nothing to go stale

One GPU, one machine, and for now exactly one tool that needs it: a local model doing GPU-bound
work a couple of times a week for one project that calls it. Small enough that nothing had broken
yet. Also the moment a design decision is cheapest to get right, before real usage patterns build
up around whatever gets chosen first.

The proposal that came my way was the standard answer to "two things might want the same resource
at once": a lock file. Before using the GPU, claim it by creating a marker. Before finishing, remove
the marker. Anything that shows up while the marker exists waits its turn. It's a well-understood
pattern, and the version proposed was already a careful one: atomic creation so two claims can't
race each other, enough content in the file to tell a genuinely stale lock from a live one.

What a lock file is actually for

I looked closely at why a lock needs to be careful in the first place: it isn't protecting against
the resource being busy. It's protecting against the lock itself lying. Every piece of a well-built
lock's design — atomic creation, a check for whether the lock has gone stale, cleanup on every exit
— exists to defend against failure modes the lock introduces just by existing. A resource that was
never contended in the first place doesn't need any of that. The contention was never the hard
part. The bookkeeping was.

Don't keep a record, ask

The hardware itself already knows exactly what's using it, right now, with no possibility of going
stale, because it isn't a record of a past claim. It's the current state, queried live. A single
command asks the driver directly: what's running, how much memory is it holding. No file to
create, none that can outlive the process that made it, no PID to check against a process table
that might have recycled it by now.

And it fits the actual failure this machine has had: not two processes racing to grab the GPU at
the same instant, but one process hanging without dying and quietly sitting on memory the next job
needs. A lock file wouldn't catch that at all. A hung-but-alive process holds its lock exactly as
validly as a healthy one. Asking the hardware catches it immediately, because the memory is either
there or it isn't, whatever any marker file claims.

What a lock file has to get right:

• Create the marker atomically, so two claims can't race
• Remove it again, on every path out
• A holder that died without removing it
• A process ID the operating system has already reused
• A holder that hangs without dying, and keeps the lock

What a live query has to get right: nothing. There is no stored claim, so there is nothing that
can disagree with the hardware.

Four of the five are obligations the lock creates for itself. The fifth, creating the marker
atomically, guards a simultaneous race, which is not the failure this machine has actually had.

The one thing a lock file offers that a live query doesn't — a clean error message naming who's
holding the resource — is available from that same query. The command that shows what's running
also shows which process ID owns it.

Zero staleness handling needed, because there's no state to go stale. Not a cleverer lock, no lock at all.

A second cut, caught by asking someone to disagree

A related piece of the same proposal suggested retry logic: if the resource is busy, wait and try
again automatically. Reasonable-sounding, and it was cut too — by an independent review pass
commissioned alongside the decision itself, brought in to look for problems rather than confirm
the plan.

An unattended process retrying its way onto shared hardware is precisely what this project's own
rule against unattended execution already exists to prevent. Elsewhere in the same document, the
proposal that suggested the retry logic had made that exact argument in a different context.
Nobody had done anything dishonest. It's an easy inconsistency to write past when you're the one
building the argument in the first place. The retry logic and the no-autonomous-execution
principle were reasoned about separately, minutes apart, and never checked against each other
until someone whose only job was to do exactly that.

What the proposal asked for:

• If the hardware is busy, wait and try again
• Automatically, unattended, with nobody watching

What the same document ruled out:

• Anything that can fire on its own
• Anything that runs with no person present to catch it

One document, two conclusions, minutes apart. Side by side they are incompatible; read in the order
they were written, neither one looks wrong.

Fail fast instead. If the resource is busy, say so and stop. A human decides when to try again.

What made the lazy answer the right one

None of this came from my distrust of locks in general, or a rule that says avoid new machinery. It
came from asking what problem the proposed fix was solving, and finding that its complexity existed
only to manage its own side effects. The resource was never the source of the risk. The simplest
version of the design didn't win by being simpler. It won because it was more correct, for the
failure that had actually happened, and had nothing left over to go wrong.

---

Originally published at thekilted.dev/nothing-to-go-stale.