What we shipped on 2026-07-15
Today's most interesting fix wasn't a fix at all -- it was an autopsy. The issue (poindexter#757) proposed adding a lease_expires_at column plus a heartbeat to distinguish a stalled worker from a dead one. Investigation found we didn't need any of it: TasksDatabase.heartbeat_task already existed, built specifically to refresh pipeline_tasks.updated_at so sweep_stale_tasks could tell "actively processing" from "worker died mid-stage." It just had no caller. Its only caller, TaskExecutor, was deleted whole in the 2026-05-16 Prefect cutover when content_generation.py took over, and nobody re-wired the heartbeat into the new flow. Every stale-sweep decision since has been judging purely by claim-time age (PR #2528). That's the kind of bug that doesn't throw -- it just quietly makes the wrong call for weeks, and the fix was re-wiring a call site, not inventing a schema.
The social pipeline had its own version of "no error, wrong outcome." social.generate_drafts was catching generation exceptions with a log-only except -- no re-raise, no operator alert, no persisted failure row -- so a transient LLM error left zero social_post_drafts rows while atom_runs still reported status='ok'. RetryFailedSocialDraftsJob couldn't touch it because there was never a failed row to retry. We confirmed 2 of the last 6 published canonical_blog posts had silently lost their social drafts this way. Both text-platform and per-subreddit paths now call notify_operator -- batched once per run for Reddit rather than once per subreddit -- and we added BackfillMissingSocialDraftsJob plus a poindexter social backfill CLI command to reconcile published posts against their template's graph_def, safe to run repeatedly since the atom's own existing_draft_keys() idempotency guard makes a fully-covered post a cheap no-op (PR #2532).
Test suite got its own ghost-in-the-machine treatment: nest_asyncio.apply() permanently monkeypatches asyncio.run process-wide with no unapply, so a single test faithfully reproducing Ragas's reentrant-loop bridge poisoned every later asyncio.run() call in the same pytest process -- breaking a revalidation test's "two calls, two distinct event loops" assumption, but only when run as part of the full suite, never in isolation. Fix was _isolate_nest_asyncio_patch(), a snapshot/restore context manager scoped to the offending test (PR #2529). Findings dashboard had a smaller but similarly sneaky split: emit_finding()'s contract says severity "warn", but nine call sites had drifted to "warning" over time, fragmenting one severity tier into two badges -- WARN·1172 next to WARNING·38 -- even though the router already treated them as equivalent for routing. Normalized at the write boundary now; live-data check confirmed 1172+38=1210 raw merges to 1211 after the one more that landed mid-check (PR #2526).
Smaller stuff rounded out the day: get_logger() finally got an explicit StructuredLogger Protocol return type instead of leaving Pyright to infer Any | Logger, clearing 77 false-positive reportCallIssue errors across five files (PR #2527); the validation-failures endpoint got offset pagination (PR #2522); and we pulled _llm_semantic_pexels_query out of ImageService into build_semantic_pexels_query(topic, *, site_config) -- a pure function masquerading as a method, the first real cut against a file that's grown to 1,345 lines despite three rounds of "we should shrink this" (PR #2524). All of it rolled into 0.104.0 (PR #2625), alongside a new ProbeDisabledCapabilitiesJob for surfacing silent opt-in features and per-niche cadence targets wired into the SLO probe.
None of these were glamorous fixes. The heartbeat one is the one we'll remember -- proof that sometimes the right fix isn't the one the issue asked for, it's finding the mechanism that was already built and just never got its wire connected.
Auto-compiled by Poindexter from today's commits and PRs. See the work: github.com/Glad-Labs/poindexter.
Top comments (0)