Thirty minutes and change after a scheduled writing job took the lock, a foreground session asked the gate for the same GPU, saw it busy, waited zero seconds, fell back on the spot, and the fallback lane handed it 429s. Gate, config, router: each layer did what it was configured to do, and the session that wanted the GPU got nothing for its trouble. That shape dodges every check you'd normally trust, because the lock was working, the fallback was working, the logs were clean, and nobody got served.
Two claims come out of the incident. Instant fallback moves contention to the fallback lane, so when that lane is full, the fallback is the second outage. And a lock that spans whole agent turns gives any scheduled job tenancy over the machine, because one long background turn can outlast every foreground session that wants in. Both mechanisms shipped separately, both passed their own tests, and the failure lived exactly where they touch.
The gate was doing its job
The design from my earlier single-lane work holds: one agent turn at a time on the GPU, enforced with a whole-turn flock on the managed port 8325. Whole turn means prefill, decode, retries, everything the turn does inside the agent runtime, so a request takes seconds and a turn takes as long as the model needs. A scheduled job is a turn too, and that's the part nobody designs for on purpose.
On October 24 the writing cron's session id carries an 18:00:07 fire time, and it held the whole-turn flock for over 30 minutes per the incident log. At 18:30:29 agent.log shows the gate reporting busy to a foreground session, immediately followed by fallback activation. The foreground session had model.messaging_lock_wait_seconds=0, so there was nothing to wait through: zero seconds of patience, reroute on contact. The router sent the fallback requests to the Strata lane, which was under GLM priority, and they came back 429. router.log has that line. The sidecar log pins the cron owner. Three layers, three clean log lines, and the foreground session's request never landed.
Three mechanisms, one bad composition
None of the three layers lied. The gate saying busy was accurate. The zero-wait setting was carried over from earlier work: reroute on contact, queue never. GLM priority in the router was a fix from earlier work. Each piece passed its own spec. The composition is what failed, and it's the earlier GLM priority fix that exposed the still-enabled whole-turn fallback mechanism.
The fix ran in five places
First flag: model.single_session_lock went false, after a dated config backup, and the model's 500000 context stayed untouched.
The gate patch exempts exactly the localhost lanes: 8320, 8321, 8325 and 8329 on 127.0.0.1, in acquisition and in the fallback busy probe alike. Direct gates from other machines to the box stay on; cross-machine contention is the case the gate was built for. To find every touchpoint I searched the runtime's consumers, turn_context acquire, chat_completion_helpers fallback probe and turn_facade release, then ran the suite: 29 tests passed, 12 of them actual held-flock cases. A controlled gateway SIGTERM and restart loaded the patch. No lockfiles deleted.
On the box, the launcher exports STRATA_FAIR_SLICE_S=8, so the built-in engine yields long text generations to queued requests, FIFO, with image prompts excluded by upstream code. GLM launch failures no longer invoke restore_lanes, and readiness checks demand the exact model with positive context.
The admission router applies GLM priority during startup and recovery as well as when the resident is healthy, and a wrong model can't satisfy readiness. Lifecycle control is bounded at 420s, the storage check still runs before any launch, and there's no cloud route in this chain.
Then there's the recovery wrapper, glm-recover-wrapper.py. It reclaims only idle known GLM wrappers running at max_context 0, refuses busy, queued and unknown owners, sends SIGTERM to the exact config launcher and waits 65s. No forced kill anywhere in it. Cold startup and healthy wrappers are untouched, the router calls it after the storage check and the GPU eviction guard, before deciding between an existing launcher and a new one. Four isolated guard tests passed, no production crash was injected, and the healthy branch got verified live.
After the numbers landed
A concurrent test ran a short request and a long 1024-token request together under the 8-second fair slice. The short one, flagged FAIR_OK, finished at 10.5s, before the long one landed at 23.1s, and the launcher's fair-slice logs confirm the order. A real agent run through the managed port came back MAYA_GLM_OK after the reload and again after the router activation, fallback None. Five router tests passed. The GLM health endpoint reports the full 500000 context. What I haven't run: a full-context and image-contention stress test, so the fairness numbers cover text workloads only.
The incident also meant housekeeping. I paused the scheduled writing job, revoked the current fire's claim, and restarted the gateway gracefully to load the patch and end the old cron holder. The schedule came back for the next morning's fire and the interrupted run wasn't re-fired. The controlled GLM restart used the same pack, the unused rescue copy stayed where it was, and the hardware cause of the earlier SSD controller dropout is still unresolved, so the box carries no forever guarantee.
The lesson I'm keeping: contention you route around is contention you still pay for, it just stops appearing on any metric you watch. A zero-wait fallback turned the contention into 429s, and neither the hold nor the refusal ever showed up as a queue depth anywhere. If you run fail-fast against a shared resource, ask the gate what happens to the session that arrives during someone else's whole turn, then ask the fallback lane what happens when it's full.
What does your fallback lane do when the fallback is the outage? I want to read the designs, comments are open.
Top comments (0)