I shipped a model switcher last week. A settings screen, two local GGUF models on disk, a "Make active" button. I clicked it, watched a spinner for about ninety seconds, and then my Mac rebooted.
Not the app. The Mac.
panic(cpu 0 caller 0xfffffe004ac0433c): watchdog timeout: no checkins
from watchdogd in 93 seconds (299 total checkins since monitoring last enabled)
...
Compressor Info: 12% of compressed pages limit (OK) and 8% of segments
limit (OK) with 7 swapfiles and OK swap space
A kernel panic is the operating system filing a formal complaint. This particular flavor — a watchdog timeout — is not about my code crashing. It means userspace as a whole stopped responding, and after 93 seconds of silence the kernel concluded the machine was beyond saving and pulled the plug itself.
My settings button did that.
What the watchdog actually saw
watchdogd is a tiny daemon with one job: telling the kernel "userspace is still alive" every few seconds. It doesn't do anything heavy. For it to miss checkins for a minute and a half, the system has to be starved so badly that a trivial process can't get scheduled or can't allocate a page.
The panic log has the tell: 7 swapfiles, and a memory compressor reporting "OK" with plenty of headroom. The machine spent its final minutes frantically paging out everything that could be paged. It wasn't enough — because the memory that mattered couldn't be paged at all.
Two models, one process
The arithmetic is embarrassingly simple in hindsight. My app runs llama.cpp in a Python sidecar (llama-cpp-python 0.3.33), all layers on Metal, on a 24 GB Apple Silicon machine. The active model was a 12B at Q4_K_M — 7.4 GB of weights, call it ~9 GB resident with the KV cache. The user switches to a 14B at Q5_K_M — 9.8 GB of weights, ~11 GB resident.
The switch endpoint did the obvious thing:
self._llama = None # release the old model
self._llama = Llama(new_path, ...) # load the new one
Drop the reference, load the replacement. It works in every tutorial, because every tutorial has one model and enough RAM.
Here's what actually happened: 9 GB of "released" old model + 11 GB of new model loading + macOS + my app + a browser, on a 24 GB machine. And the crucial detail: model weights on Metal are wired memory — physical pages pinned so the GPU can address them. Wired pages don't swap. They don't compress. When wired allocations eat the machine, the kernel has nothing left to steal, userspace grinds to a halt, and the watchdog does what watchdogs do.
"Released" is not "returned"
That self._llama = None line does less than it appears to. On our stack — llama-cpp-python 0.3.33, Metal backend — we had already observed this once in a different corner: our idle unloader drops the model object after a few minutes of inactivity, and the process footprint... doesn't come down. The Python object dies; the wired pages stay attached to the process. The only event we've found that reliably returns that memory to the OS is process exit.
I won't claim this is a law of nature. Maybe an explicit close() at exactly the right moment behaves better in your version, maybe a future release fixes it. But we measured ours, twice, on the machine that matters: memory comes back when the process dies, and not before. If your swap design depends on the old model's memory being available for the new one, you're betting your users' machines on a deallocation you don't control.
The fix is boring, and that's the point
The switch button no longer touches the loaded model at all. It writes the chosen model path to a small state file and returns {"restart_required": true}. The app then restarts the sidecar process: the old process dies (taking every wired byte with it, guaranteed, by the only mechanism that guarantees it), the new process reads the state file and loads the chosen model.
Cost to the user: 10–20 seconds of "Restarting the server…" instead of occasionally costing them their entire machine. I'll take that trade every day of the week.
Three traps between me and the boring fix
1. terminate() doesn't wait. Signalling the old process and immediately launching the new one reintroduces the exact bug through a different door — for a few seconds both processes hold their models. You have to wait for actual death: SIGTERM, poll until the process is gone, SIGKILL after a deadline (a load stuck inside C code never reaches a signal handler). Only then launch.
2. The port lies after death. Our supervisor probes the sidecar's port before launching, with a plain bind(). The connections a just-killed process leaves in FIN_WAIT_2 make that probe fail — port "busy" — so the supervisor helpfully relocated to the next port on every single swap. The server itself binds with SO_REUSEADDR and retakes the port without complaint. The probe now does the same, plus a connect() check — because SO_REUSEADDR alone will happily bind 127.0.0.1 right over someone else's wildcard listener, and shadowing a stranger's server is worse than moving.
3. Gate by total RAM, not available. We added a pre-flight check: refuse to load a model that can't fit. First version used available memory. Wrong metric — at swap time the old model is still resident, so "available" is tiny and the gate refused every legitimate swap. Whether a model fits this machine is a property of total RAM (the weights are wired; the OS will evict everything else to make room). "Available" is only good for a soft warning: close some apps, this will be tight.
If you're building a model switcher
- Assume wired model memory returns on process exit and at no other time. Design the swap as a restart.
- Wait for the old process to actually die before starting the new one. Poll, then SIGKILL. No overlap, ever.
- Probe ports the way your server binds them (SO_REUSEADDR), and verify with connect(), or enjoy your app quietly migrating ports.
- Hard-refuse models by total RAM. Warn by available. Never block on available.
- If the refusal message says "needs ≥N GB", compute N from the actual numbers. Users on a 24 GB machine reading "requires ≥24 GB" will have questions.
The repro script and the full panic log are here: https://github.com/JackYU96/swap-models-restart-process
The Mac survived. The hot-swap didn't.
Top comments (0)