My agent runs several sessions at once against one local model. A chat turn, a cron job, a subagent mid-scan, they all point at the same Qwen backend behind llama-swap on :8080, one model loaded on one 3090. Two turns hitting that engine together thrash the KV cache the first one is using, so the rule became: whoever is generating owns the lane, everyone else takes the cloud fallback. The ownership has to hold across processes, because my sessions aren't one process. The chat channel, the cron runner, and the subagent tool each spawn their own, and a lock that lives inside one Python process protects nothing when three of them are the problem.
One lock, held per turn
The gate is a cross-process flock, one lock per backend, keyed on host and port. The scope of the hold was the first design call that mattered. Hold it per session and one long-lived chat session owns the 3090 forever while every cron job starves, so the hold is per turn. Acquire at the start of a turn, release when it ends, so between turns the lane is free and whoever generates next walks straight in. A quiet session is just a quiet session, generation is the only thing that holds the lane.
The second call was cleanup. flock lives in the file descriptor, so the kernel drops the lock when the process dies, and that kills a whole category of code I didn't have to write. No stale-lock reaper, no heartbeat file to keep fresh, no guessing whether a 20-minute-old lock is dead. My release path can crash as hard as it wants, because the gate still reopens. Next to the lock sits a small sidecar with pid, session, model, and timestamp at acquire time, and that file is what makes the gate inspectable when something looks stuck.
Three hook points, fail open everywhere
The gate hooks the agent loop in three places. At turn build, right after the runtime restores the primary model, the turn tries to acquire, and a busy gate sends it down the actual fallback chain, which starts with the cloud model. Then the fallback walk itself skips chain entries pointing at a busy local backend, so a session on a cloud-primary profile can't fall back into a saturated engine. And release lives in the outermost finally of the turn runner, so a turn that finishes, errors, or gets cancelled releases in the same place. There's also a new failover reason, local_backend_busy, and it renders as "local model in use by another session", so a deflection looks like a deflection and never like a provider outage.
Fail-open was non-negotiable. This gate is a performance guard, agent availability never depends on it, gate errors don't block a turn, no config means behavior exactly like before, and anything non-local skips the gate entirely. The worst case of the gate itself breaking is two turns sharing the card, which is the world before the gate.
The tests passed, the duel didn't care
The suite went green on the first run, 17 gate tests covering mutual exclusion, the busy-skip, fail-open, and a real cross-process flock case, plus 74 regressions across the fallback and turn-lifecycle code. Then I ran the version that actually proves something, two live CLI sessions and a cron turn, and the first duel caught a bug the suite structurally couldn't. A flock only excludes processes that compute the same lock path, and a unit test runs one process with a home directory I hand-select, so any internally consistent path passes it. My profile-home stripping resolved the wrong root, the lock and its sidecar landed under the per-profile tree, and the canonical shared dir sat empty, which I only saw because the duel prints where the lock file actually lives. The deflection still worked between the two CLI sessions, they were the same profile and agreed on the same wrong path, and that's the sneaky part. Sessions come from every profile on the box, the shared root exists so that every one of them lands on the same lock, and a gate that only some processes can see is two gates.
One root-fix commit later, duel two ran with the lock and sidecar at the canonical path, B deflected to the cloud model, A released clean, pass. Duel three added the probe I actually wanted, the lock file free after A said done, and a fresh acquire proving the local model was immediately re-acquirable.
The way back is automatic
A fallback that never returns is an expensive way to run one model, so the return trip got its own check. The primary runtime restores at the top of every turn, so the gate only deflects turns that collide, and the next scheduled turn tries the local model first again. The live proof is one cron turn across one night, local model at 00:05, deflected to the cloud at 00:11 while a duel held the gate, then back on the local model at 00:17, with no intervention anywhere in between.
What shipped
The flag is on in both of my agent profiles, model.single_session_lock: true, the gateway came back up on the fixed code twice that night, and every channel reconnected clean. The gate keys on the backend rather than the model, because the card loads one model at a time anyway, and the day a second GPU lane shows up the key becomes model-scoped.
If you run one model server behind anything that spawns concurrent sessions, you want this shape. Spend the effort on two real processes doing real turns, because a single-process suite will pass no matter where your lock lands, and fail open, because a gate that can deadlock your agent is worse than the contention it was built to stop.
One 3090, one model at a time, September 2026.
Top comments (0)