What we shipped on 2026-07-29
We found a particularly nasty class of bug in fix(video): mount the demo-clip dir on worker AND prefect-worker (PR #2912). We'd wired cli_demo into the shot list as a first-class source (PR #2909), but we forgot to mount the demo_clip_dir on the actual workers. The terrifying part was that the video renders still succeeded; they just fell through the backfill ladder and rendered branded cards instead of footage. It looked like a working video, but it was empty.
The demo-clip lane is now fully operational, from recording real CLI footage as clips with VHS (PR #2897) to adding poindexter logs for better visibility into the process (PR #2901). We've also added pitch-preserving atempo to our TTS (PR #2896), which finally lets us tune cloned-voice pace without warping the pitch.
On the GPU side, we discovered another silent failure during our P1 soak. Our gpu_evictable_process_pattern was looking for the substring "ollama", but on Linux, the exporter labels the process as llama-server (PR #2905). The eviction credit was 0.0 for the entire soak, and because the fit gate fails open on missing telemetry, we never knew it wasn't working. We fixed the pattern and moved forward with P2 group 1 migration, implementing bounded GPU waits for our QA rails (PR #2908). By forwarding max_wait_s and priority through dispatch_complete, Ragas and DeepEval judges can now wait in a background priority queue rather than burning the full lock ceiling.
We also had to deal with alert fatigue. Our boot-time model validator was crying wolf on 16 "missing" models every start, when only one was actually missing (PR #2911). It turned out we were using an unescaped _ LIKE wildcard in our queries and lacked structural rules for non-Ollama namespaces. We tightened the logic so the checker is actually trustworthy again.
A few other stability wins: we stopped Chatterbox from squatting VRAM when idle (PR #2904) and fixed a log line that under-reported the reclaim process (PR #2907). We also performed an AST sweep of our LLM dispatch sites to find where phase was omitted (PR #2910); we found that podcast_service was effectively truncating its own long-form completions because it inherited a global 8192 window.
We wrapped these up into release 0.113.0 (PR #2916). Today was a reminder that "it didn't error" is often the most dangerous state a system can be in.
Auto-compiled by Poindexter from today's commits and PRs. See the work: github.com/Glad-Labs/poindexter.
Top comments (0)