The Guardrail Has to Be Code: How a Runaway Local LLM Corrupted APFS and Bricked a Mac Mini

Originally published on hexisteme notes.

I run a small fleet of local agents on a Mac Mini M4 with 24 GB of unified memory. One afternoon it went down hard — no wake from sleep, no boot chime past the logo, and in recovery mode an APFS volume that refused to mount. The unsettling part: I hadn't asked it to do anything heavy. Something had quietly started a large language model in the background, on a disk already nearly full, and by the time I noticed, the filesystem was past saving. Recovery took more than a day. This is the postmortem — and the fix, which turned out to be a single False and a checklist, not a note to myself.

A background LLM server, built on Apple's MLX, loaded a 14B and an 8B model on a Mac whose disk was already nearly full. Unified memory overflowed; macOS tried to swap; the disk had no room; the failed writes became I/O errors, and the boot volume's APFS metadata came out corrupt enough that the machine wouldn't boot — a 24-hour recovery. The durable fix was not a "don't run this" note — a note is a soft default, and defaults get flipped. It was a code-level hard block, plus a revival checklist. The lesson underneath: on a unified-memory Mac, free disk space is your memory safety net.

The timeline: from "a little slow" to "won't boot"

It didn't look like a memory problem — just the machine getting sludgy: beachballs, slow app switching, the stuff you blame on too many tabs. Then the display went black, gone. A forced restart got the Apple logo and a progress bar that stalled and stayed. In recovery mode the shape came clear: Disk Utility saw the container, the data volume wouldn't mount, and the repair pass — fsck_apfs — hit metadata it couldn't reconcile, a volume whose own consistency layer had lost consistency. Repair couldn't finish, and the fix cost the better part of a day between attempts and a restore.

The root-cause chain: disk pressure is what turns OOM into corruption

Reconstructed after the fact, the chain had six links, none exotic:

• The disk was already low on free space — the precondition that makes the rest dangerous, fine until the OS needs room and there isn't any.
• A background server loaded large models — a 14B and an 8B, several gigabytes of resident memory with no window open.
• Resident memory exceeded physical RAM. macOS doesn't refuse this; it manages it.
• macOS reached for swap — compress, then page the overflow to swapfiles on the boot volume. The escape valve needs disk room.
• The swap writes failed — no room to grow the swapfiles, so page-outs returned I/O errors.
• The APFS metadata didn't survive it — left inconsistent, and an inconsistent APFS volume won't mount.

I'll flag that last link, the one I can't fully prove. APFS is copy-on-write and crash-consistent; a full disk alone should give clean ENOSPC errors, not corruption. The honest reading isn't "disk-full corrupts APFS"; it's that a nearly-full disk under sustained failing swap writes is a rare corner where the filesystem's guarantees get tested at the worst moment, and mine didn't hold. The rule doesn't need the exact trigger: never let the OS be forced to swap onto a disk with no room. You defend that corner by never entering it.

Why a note in the config was never going to hold

The detail that decided the fix: I didn't start that server. A global config had a line of guidance that local model delegation was preferred for background chores; some path read that as license and auto-started it. I was never in the loop. That's why the fix couldn't be another note. A soft default gets flipped by things that don't read it — an environment variable exported in another shell, an auto-start script, an agent following the global rule over the local exception. If a code path can physically brick the machine, the guardrail can't be a sentence asking it not to. It has to be code that can't.

The fix: make the dangerous path refuse to run

So I replaced guidance with refusal, at every layer that could start the server — defense in depth, independent blocks, each alone enough to stop the process. Across the two repositories that could spawn the model:

• One constant as the source of truth — a single _MLX_ENABLED = False every layer reads.
• The router's health check is wired to fail. _check_local_server_health() returns False unconditionally, so decide_router() has no branch that can select local.
• The executor escalates first — its entry point returns {"escalate": true} and exits before importing mlx_lm or opening a socket.
• The spawner is stubbed, original preserved — the command that used to nohup mlx_lm.server ... now refuses; the real one is renamed with a _LEGACY_ prefix, kept out of the live call graph. The dormant server manager and batch engine got the same treatment.
• A companion constant in the other repo, so neither codebase can start what the other one blocks.

_MLX_ENABLED = False              # one source of truth; every layer reads it

def _check_local_server_health() -> bool:
    return False                  # router has no branch that selects local

if __name__ == "__main__":        # executor: escalate & exit before the
    print(json.dumps({"escalate": True}))   # heavy import or any socket
    sys.exit(0)

def cmd_start(*args):             # CLI start refuses...
    raise SystemExit("local model server disabled — see revival checklist")

def _LEGACY_cmd_start(*args):     # ...original kept intact, out of the call graph
    ...                           # `nohup mlx_lm.server ...`

Every block is a positive refusal, not a missing feature: it runs, and says no. A missing feature invites re-adding; a refusal is a decision you must consciously reverse — and because the _LEGACY_ originals are right there, reversing it is a rename, not a reconstruction.

The other half of a hard block: a revival checklist

A hard block with no documented way back is its own trap — it calcifies into dead weight, or gets ripped out in a hurry without restoring what made it necessary. So it ships with its inverse, an explicit ordered checklist to re-enable it, deliberately:

1. Verify disk headroom first — df -h / must show 20 GB+ free before anything competes for memory.
2. Re-check iogpu.wired_limit_mb — so a model can't wire away the memory the OS needs.
3. Rename the _LEGACY_ functions back, restoring the original call graph.
4. Restore the second repository in lockstep — its constant, its server manager, its batch engine.
5. Re-enter with the smallest model — a 3B 4-bit model first, confirm the machine stays healthy, then scale up.

The checklist turns "turn the LLM back on" from an impulse into a gated act: the block stops the machine bricking today, the checklist stops a careless un-block tomorrow.

The rule the whole thing taught: disk free space is a memory safety net

Step back from the specific server and the failure has one transferable lesson, about the hardware more than the software. On Apple Silicon, memory is unified: CPU and GPU share one physical pool, no separate VRAM. That pool is 24 GB, and everything draws from it — the OS, every app, any ML workload. A model's weights come out of the same 24 GB the OS is trying to live in. (The iogpu.wired_limit_mb sysctl caps how much of the pool can be wired for GPU/ML use — set it too high and a model can starve the OS.)

And macOS backs memory pressure by swapping to the boot volume — so free disk space isn't just for files; it is the runway the memory system needs when the pool overflows. When that runway is gone, "out of memory" becomes a filesystem-integrity problem. Treat free disk space as reserved memory: below a comfortable buffer — 20 GB is my floor — you are risking not a full disk but the swap subsystem, and through it the filesystem. That produced three standing rules, all aimed at never reaching overcommit:

• One heavy job at a time. On 24 GB of shared memory, two 14 GB-class workloads don't coexist, and neither does one heavy ML task alongside Photoshop, a video editor, and a VM. Light work is free; simultaneous heavyweights are banned.
• A memory preflight before generative work. Sum the genuinely available memory — free + inactive + speculative + purgeable pages, which vm_stat reports — and if it's under roughly 8 GB, close apps first.
• Low-memory mode by default, and 20 GB+ of disk free as a standing condition — the buffer only helps if it's there when the pressure hits.

The recurrence that proved the rule

About two and a half months later, I got to run the experiment again by accident: a heavy image-generation workflow at 13.7 GB, plus Photoshop, a video editor, and a VM all open. The sum blew past 24 GB and the machine blacked out — same overcommit, same trigger as before.

But this time it came back: a blackout, a hard restart, a clean boot, no corruption. The one variable that had changed was the one the postmortem told me to watch — the disk. It had 55 GB free and a healthy swap buffer, so the overflow writes succeeded. The machine fell over instead of falling apart, and I lost minutes instead of a day.

Dimension	2026-04-21	2026-07-03
What overcommitted	Background LLM: 14B + 8B models	Image workflow 13.7 GB + Photoshop + video editor + VM
Memory state	Exceeded 24 GB unified	Exceeded 24 GB unified
Free disk at the time	Nearly none	~55 GB
Swap outcome	Writes failed (no room)	Writes succeeded (healthy buffer)
Result	APFS metadata corrupt, unbootable	Blackout, clean reboot
Recovery	24+ hours	A few minutes

Same trigger, opposite outcomes; the dimension separating "brick" from "annoyance" was free disk space. So the rules got sharper — "one heavy job" now names the combination that broke it, and disk headroom went from hygiene to load-bearing invariant. The code block keeps that server from sneaking back on; the disk rule makes sure the next overcommit — and there's always a next — costs a reboot, not a filesystem.

More notes at hexisteme.github.io/notes.

Comments

[Tae Kim]

The "note vs code" distinction is the correct frame for any agent constraint: a system prompt instruction that says "check disk space first" is a note, which means it gets flipped when context pressure, tool chaining, or a multi-step plan pushes the check out of scope. The only constraint that survives a distracted or context-limited model is one enforced at the process boundary before the model code runs. For memory-sensitive workloads that means the wrapper checks free space and rejects the load call outright - same as your checklist, but automated and unskippable. The principle generalizes: if a constraint matters enough to write down, it matters enough to make unbypassable in code.