What we shipped on 2026-07-27
We almost shipped a blind scheduler. We'd built the P0 observe layer for the GPU scheduler to track lease stats and queue mirrors (PR #2836), but when we went to verify per-process VRAM rows, nvidia_gpu_process_memory_mib was serving header-only forever. The culprit was a classic container pitfall: the exporter lived in its own private PID namespace, so nvidia-smi --query-compute-apps returned zero compute apps while the host listed five (PR #2838). We had to switch the service to pid: "host" just to see what was actually happening on the cards.
Once we could actually see the processes, we hit a cardinality explosion. The Linux drivers were reporting the full command line as the process_name, meaning labels like process="Claude --gpu-preferences=UAAAAAA..." were being minted for every single run (PR #2839). We had to implement a strip-and-basename derivation in the exporter to collapse those into clean identifiers like claude-desktop or ollama so our metrics didn't melt.
With the telemetry finally honest, we landed the P1 queue admission and wait contracts (PR #2837). The core logic now lives in services/gpu_admission.py, where a decide() function manages an ETA gate--calculating budgets using p90 ā elapsed from our new gpu_lease_stats table--and a fit gate that checks if the estimate fits within free VRAM minus gpu0_headroom_gb. We've kept this doubly inert behind a gpu_sched_enabled flag and zero production call sites for now, but the machinery is in place.
We also spent some time cleaning up the "noise" in our CI pipeline. We realized that no workflow had ever actually run a Python linter; npm run lint:python existed, but it wasn't enforced (PR #2841). In the process of gating Ruff, we found that infrastructure/grafana/scripts/optimize_portrait.py had sat unparseable for two months because a public-mirror strip had re-indented code into column 0 back in May. We also fixed a colliding namespace issue where our # noqa: adapter-ok marker was triggering "Invalid # noqa directive" errors on every run, moving it to the simpler # adapter-ok: form (PR #2840).
The rest of the day was about resource survival and safety. Chatterbox narration was losing the VRAM fight on both cards, so we moved TTS execution to the CPU (PR #2852). We also added verify-recovery to our backup system to prove that offsite repos can actually open without depending on the local machine's state (PR #2852).
The GPU scheduler is now a reality in the code, even if it's still sleeping. From here, we just need to migrate callers group by group and see if our PĀ² streaming quantiles actually predict wait times in the wild.
Auto-compiled by Poindexter from today's commits and PRs. See the work: github.com/Glad-Labs/poindexter.
Top comments (0)