What we shipped on 2026-07-17
We spent today cleaning up "ghost" bugs--the kind of silent failures that don't trigger alerts but hollow out your data. The most jarring was discovering that lab and writer observability had been dead since June (PR #2649). We found 6,243 rows in atom_runs.metrics that were completely devoid of content_length, model_used, or prompt_template_key. The culprit was a shim in services/atom_registry.py passing record_sink=None to make_stage_node, which effectively threw away all computed metrics before they could be persisted (PR #2649).
This pattern of silent failure extended into our LLM provider. While verifying other changes, we realized LiteLLMProvider.embed() was never passing the api_base to litellm.aembedding (PR #2657). Because it fell back to a default Ollama endpoint at localhost:11434, every single embed call failed inside the worker container while the logs stayed deceptively quiet (PR #2657). We also caught our QA rails lying to us; self_consistency_rail.evaluate() was returning a score of 1.0 even on skip paths, meaning "didn't check" and "perfect" were indistinguishable in the audit log (PR #2655).
On the security front, we finally stopped the Bandit linter from acting as a GitHub-issue firehose (PR #2650). After triaging 91 issues and finding almost all were false positives--mostly because Bandit lacks dataflow analysis for our services/ patterns--we moved to a ratchet system. We grandfathered existing findings into scripts/ci/bandit_baseline.json, so CI now only blocks net-new findings (PR #2650). We tightened that baseline from 61 down to 38 findings after burning down the B608s (PR #2651) and pinned HF model revisions across our image and video servers using from_pretrained and hf_hub_download to close a supply-chain exposure (PR #2658).
We also dealt with some regressions in the media pipeline. A previous rename of sdxl to image_gen had been shipped without a backcompat shim, causing older frozen shot lists to fail validation (PR #2652). We found one production task that had burned through 304 checkpoints--re-paying for TTS and transcription over and over--because the media_reconciliation watchdog kept re-dispatching it into a validation failure (PR #2652). We also patched a latent defect in podcast_persist.run where an OSError during a file move returned an empty list instead of an error, masking disk failures as legitimate no-ops (PR #2656).
It was a day of hardening. By turning the linter into a ratchet and surfacing these silent failures, we're moving away from hoping the system is healthy toward actually proving it.
Auto-compiled by Poindexter from today's commits and PRs. See the work: github.com/Glad-Labs/poindexter.
Top comments (0)