What we shipped on 2026-07-31
We spent a chunk of today wrestling with the cost of silence, specifically in our media pipeline. A 2,177-word article had cleared every QA rail and rewrite pass only to stall indefinitely in the media block behind a render (PR #2921). Across five attempts on two tasks, we burned $0.50 and 278 LLM calls without publishing a single word because the system just hung until a 30-minute stale-reclaim reset it. We fixed this by implementing bounded GPU waits for fail-soft media stages (PR #2921), setting gpu_sched_media_max_wait_s = 120s based on our recent lease stats.
While fixing the hangs, we found a class of bugs that were completely invisible to our tests because our test harness supplied its own plumbing rather than using the actual deployment config (PR #2922). The poindexter media demos bake command was writing to /tmp/poindexter-demo-clips while the system was looking for files in demo_clip_dir. Everything reported success--the bake, the render, the CLI--but the resulting videos were just empty. We've reconciled this through the SiteConfig DI seam (PR #2922).
The Chromium sandbox presented a second hurdle: VHS requires seccomp=unconfined to function, but granting that to our long-lived worker would permanently widen the attack surface for any container handling external LLM output. To solve this without giving up security, we moved the demo clip baking into a throwaway sidecar triggered by a host systemd timer (PR #2924). This keeps the privilege grant scoped to a container that lives for roughly 60 seconds and runs only read-only CLI commands. We added test_long_lived_services_do_not_relax_seccomp as a tripwire to ensure no one moves this back into the worker for "simplicity" later (PR #2924).
We also closed a critical gap in our CI/CD pipeline where we were scanning Dockerfiles with Trivy but never actually building the images (PR #2920). This led to a genuine divergence between host and image; inside the worker, importlib.metadata.entry_points() was returning 0 while the host venv reported 26 jobs. We now build build-brain on ubuntu-latest and build-worker on our self-hosted runner (PR #2920).
On the console side, we hunted down a RangeError: Maximum call stack size exceeded that had been lurking in console-unit (PR #2919). The culprit was loadAllSettings aliasing its response array--the accumulator was a reference to page 1's array, and as the loop mutated it, the array eventually pushed itself into itself. Combined with using the spread operator for large pages, we were blowing the argument limit (PR #2919).
Finally, we discovered that our Dependabot holds for TypeScript were being ignored entirely (PR #2925). Despite a merge to hold major updates, Dependabot bumped typescript from 6.0.3 to 7.0.2 just twelve hours later. We've abandoned the group-based filters and switched to blunt per-dependency versions: ranges for both typescript and @types/node (PR #2925).
The system is tighter now, though we're still wary of how many "invisible" failures can hide behind a successful exit code. From here, our focus shifts back to the content throughput now that the media block isn't a black hole for budget and time.
Auto-compiled by Poindexter from today's commits and PRs. See the work: github.com/Glad-Labs/poindexter.
Top comments (0)