The layer between your phone and a local model treats "the engine has no capacity" as a failed request, and failed requests get thrown away. That's the wrong contract. The work isn't gone, the client just stopped paying attention to it, and once nothing holds the prompt there's no queue left to retry against. What I built after keeps every accepted message on disk until something actually answers it, and the numbers below come from that run.
What the night looked like
Oct 25 at 23:39:51, and again at 23:42:26, the router logged ServerDisconnectedError, followed by GPU ownership refusals and a Strata priority 429. The cause was mine. I'd been setting up full-duplex experiments on the box, killing several model processes by hand with sudo kill -9, and one runaway grep -rl / from that same work was still scanning eight hours later under PID 397304. So the endpoint went down while my own session was mid-turn.
Then the config lied about what was allowed. GLM priority forbids Strata taking either card, yet the fallback list still carried Strata entries and automatic ensemble references. The message hit a 429, the 429 read as quota, nothing retried, and the prompt died in transit.
What the receipts hold
Every accepted message gets written down before the model builds anything. Atomic fsync JSON per profile, 0700 directory, 0600 files, carrying the GUID, the FIFO time, attachments, and the logical source identity of the request. Startup restores messages that never started. Started GUIDs stay owned by transcript and active-turn recovery, so a restart in the middle of a turn can't run the same work twice. /stop and /new cancel receipts for that session only, other chats keep theirs. Malformed or unknown-version records stay on disk.
The waiting loop refreshes the active-turn marker every 30 seconds while prompts sit in line, and retries only the local failures that produced no effects yet. Anything already delivered to the reader stays put. Busy prompts bypass the overwrite, debounce and drop-cap behavior of the messaging adapters, but only on the opted-in local path. Cloud, background, auth, format errors and already-delivered partials keep their old bounds, and the defaults document opt-in false.
The queue survives because it knows whose prompt it holds.
Dedup keyed on the wrong thing
An end-to-end ownership probe found the second real bug, my dedup keyed on timestamp plus text. Two distinct accepted inputs with identical text at the same timestamp collapsed into one, and a queue that collapses two prompts into one keeps whichever it saw first. On 20 cases built to be distinct the original turn handler passed 4 out of 20, sixteen of twenty queued prompts eaten by a rule meant to catch replays. Keyed on input-owner identity it passes 20 out of 20.
Commit 195cf43698 keeps the timestamp fallback for legacy messages with no gateway ownership metadata, which means transcript repair adopts only that admission's committed row when an owner exists. No database change, no schema change, format version 1 stays as it is.
Ever watched two identical messages sent seconds apart come back as one reply? That's this bug, wearing a friendlier name.
The router's share
Telling the client which kind of busy this is belongs to the router. The agent loop just waits there, it doesn't decide what the engine can run. Connection failure before headers returns 503 local_provider_unavailable. Physical contention returns 503 local_gpus_busy. A model priority rule that forbids the fallback returns 409 model_priority_conflict. All three used to arrive as one fake quota code, and quota is exactly what made the queue path dead-end there. A 429 says you're out of budget, a 503 says come back, clients act on that word. No automatic retries after delivered bytes.
What I ran before believing any of it
The canonical runner takes 118 tests across 14 files covering queueing, ownership, crash recovery, busy sessions, delivery and legacy persistence. One of them is the 20-case probe run against real gateway, provider and persistence subprocesses. The router file has 9 passing, the guard plugin 5. Live fault injection wrote three temporary 503s into a test agent only, then forwarded its next attempt to the production GLM endpoint at 8325 with no fallback, same turn retained, one queue notice, result string MAYA_GLM_OK. Nobody got told about it and the engine never got killed for the test. Fair scheduling stayed at STRATA_FAIR_SLICE_S=8 and health still reported 500000 context while all that ran.
Two traps cost more than the code did. First, the reload. Restarting the named profile only reloads that profile, shared core changes don't move through it, so the first pass left the behavior broken until an explicit default-profile restart. Second, the outage itself. My agent had permission to run whatever setup commands it wanted, which made killing the engine during my own experiments a normal part of the evening, and the disconnect afterward carried no note about who caused it. A profile-local interlock now blocks numeric kill commands and named inference lifecycle, cache or GPU resets from inside the loop, diagnostics stay allowed. Maintenance requires a root-owned 0600 marker file that expires, created outside the agent's own tools. It gates whole commands at the tool boundary, shell semantics stay untouched.
Where it still doesn't earn trust
The final iMessage round-trip never got confirmed. I asked for a real test and the reply didn't arrive before I wrote this up, so treat the messaging path as designed, unit-tested, and unproven on live traffic. Physical faults may need a human with a power cycle while receipts keep the work regardless of what the disk does. A half-answer stays half until you ask again, since already-delivered partials are never replayed. And the whole-worktree lint pass carries pre-existing findings in unrelated modules, my new files clear their own health, ruff and footgun checks.
Anyone running local inference behind a messaging adapter has this fork in front of them, fail the request or hold the work. What does your queue do when the engine is mid-repair, and how long did you wait before that answer stopped being in the manual?
Top comments (1)
A busy GPU should be modeled as backpressure, not a vanished request. Durable queuing, an explicit accepted state, retry policy, and user-visible progress make overload recoverable; otherwise an agent can appear unreliable even when the model itself is fine.