What we shipped on 2026-07-12
Today's most instructive fix wasn't a fix at all -- it was admitting the last fix was wrong. fix(power): stop false PSU-floor alert storm (PR #2371) went after a real problem: the brain paging Telegram roughly 15 times a day with "No real PSU data," every page false, the Shelly plug reading a steady ~330W the whole time. Root cause: the host exporter built /metrics synchronously per request -- nvidia-smi + AIDA + Shelly poll, measured at 2.5-6.7s, sometimes wedging Prometheus to 275s -- against a 3s scrape timeout. Eighteen floor-cycles per 24h, each one a page. The fix moved collection off the request path onto a background thread.
Then the fix broke something else. fix(power): don't let the nvidia-smi watchdog kill the exporter (PR #2374) landed as a same-day follow-up: moving collection to a background thread let the old #319 nvidia-smi watchdog's SystemExit escalate via os._exit, turning a previously-harmless watchdog fatal. Three consecutive nvidia-smi timeouts under GPU load tripped it, the exporter hard-exited, the host task didn't reliably restart it, and all host metrics went dark -- caught in post-deploy verification about five minutes after #2371 shipped. The watchdog existed to kill a process wedged on the request path; #2371 removed that request path, so the watchdog had gone from "necessary" to "pure downside" without anyone deleting it. The actual fix was small: the collector loop now swallows the timeout instead of escalating it.
The same pattern showed up in QA. fix(qa): qa.vision fail-open stops mispaging a healthy vision model as unavailable (PR #2364) diagnosed a vision_scorer_unavailable false-page as a None database_service.pool and shipped a site_config._pool fallback. The follow-up, fix(qa): vision false-page is a thinking-budget truncation, not a dead pool (PR #2369), found the pool was never None -- task b740e4b8 showed qwen3-vl:30b acquiring the GPU, running 11.2s and 26.4s legs, logging thousands of tokens, releasing cleanly, and then paging "unavailable." If the pool had actually been None, dispatch never happens -- no GPU, no tokens. The page was a truncated-response artifact, not infrastructure at all. A third investigation, ci(rerank): guard the real sentence-transformers import against version-skew re-locks (PR #2370), started from a report that a huggingface-hub 1.16.1 lock silently degraded the reranker in prod -- reproduction disproved it outright, confirmed live in both poindexter-worker and poindexter-prefect-worker, and no dependency change shipped.
Three separate threads, same shape: an alarm, a plausible-but-wrong diagnosis, then the real root cause once someone actually read the logs instead of trusting the alert's framing. Not glamorous work, but it's the work that keeps the pager honest.
Underneath the corrections, real forward motion. Task b740e4b8 -- the same VRAM near-duplicate draft that triggered the vision investigation -- turned out to be a ~65-70% content re-tread of two already-published posts, and it's what motivated Content-embedding topic dedup (PR 1) + RAG-self-echo spec (PR #2359): the old word_overlap topic dedup engine is semantically blind, and the VRAM dupe scored only 0.55 on title-similarity but 0.735 on content -- a title-only gate would have burned a full ~17-minute generation before catching anything. The new ContentEmbeddingDeduplicator reuses the same pgvector find_similar_posts search the create-post guard already relies on, and content_embedding is now the default topic_dedup_engine for all installs. Its companion, feat(qa): content_originality rail -- whole-post RAG self-echo detection (PR #2375), renames qa.opening_originality to qa.content_originality and broadens the scan from opening-only to whole-post chunked, max-over-chunks -- catching the class of self-echo that a 2026-06 VRAM cluster exposed, where four posts opened near-verbatim because the two_pass writer grounds every draft on the nearest published posts. It ships advisory only; no gate-decision changes today, but it's built hard-block-ready.
Rounding out the day: fix(social): reap orphaned social drafts when content is rejected (PR #2366) stops social drafts from surviving a content rejection -- traced through the same b740e4b8 task and a second case where a post was published, promoted, then retracted two days later, leaving live promos pointing at nothing. All of it landed in chore(main): release 0.101.0 (PR #2376).
Most of today's real work was unwinding yesterday's diagnoses -- the pool wasn't None, the lock wasn't broken, the watchdog wasn't the villain it used to be. That's a slower kind of progress, but it's the kind that keeps the alerts trustworthy, which is worth more than any single feature ship.
Auto-compiled by Poindexter from today's commits and PRs. See the work: github.com/Glad-Labs/poindexter.
Top comments (0)