What we shipped on 2026-07-05
The firefighter we shipped in PR #2124 ran dead, not inert on prod. We'd built a deterministic self-healing loop--detect → act → verify → escalate--designed to hold pages and recover from alerts using idempotent actions like restart_container (PR #2124). But when it hit the wild, every alert paged as if no rules existed.
The culprit was a classic test/prod import divergence in the brain/Dockerfile. We were COPYing a flat list of files into /app/, but since brain/remediation/ is a subpackage, it never actually made it into the image (PR #2134). In CI, the repo-root was on sys.path, so everything looked fine. In prod, alert_dispatcher.py hit an except ImportError and silently bound no-op stubs. We've since updated the Dockerfile to explicitly COPY remediation/ /app/brain/remediation/ and replaced that silent degradation with a loud warning (PR #2134).
While auditing those failures, we realized our QA layer was lying to us. PR #2136 was an exercise in honesty: we discovered the dashboard looked comprehensive while only two gates actually enforced anything. We've moved to an honest gate policy where only programmatic_validator and llm_critic are hard rejectors (PR #2136). Other rails, like deepeval_brand_fabrication, have been demoted to advisory; they still provide a per-rail breakdown for the operator, but they no longer gate production. We also killed several silent failures in the publish cap and webhook paths--now, if it breaks, it fails loud (PR #2136).
On the distribution side, we moved the operator console to a Pro-tier overlay (PR #2137). This wasn't a simple directory strip; Starlette's StaticFiles raises a RuntimeError at construction if the target directory is missing, which would have crashed the OSS backend at boot. We solved this by introducing mount_operator_console(app) in utils/operator_console.py, which guards the mount based on the directory's presence (PR #2137).
We wrapped these changes into release 0.95.0 (PR #2139). Between the self-healing firefighter and the honest QA gates, the system is finally telling us the truth about its own health.
Auto-compiled by Poindexter from today's commits and PRs. See the work: github.com/Glad-Labs/poindexter.
Top comments (0)