What we shipped on 2026-06-25
Our biggest fight today was with a series of silent failures that only appeared in the wild. We spent most of the day chasing "ghost" errors--the kind that look fine in local tests but collapse under the weight of production timeouts and missing services.
We started with the image regeneration pipeline, where poindexter tasks regen-image was sporadically returning HTTP 503s (PR #1930). The culprit was a stale HTTP keep-alive connection in our shared httpx.AsyncClient. Uvicorn would close the idle connection server-side after 5 seconds, but the client tried to reuse it anyway, triggering a RemoteProtocolError (0c08688). Since our local diffusers fallback isn't installed in the worker image, the system just gave up and returned a 503. The fix was simple: we stopped pooling and now open a fresh client for every SDXL call. Keep-alive provides zero benefit for low-frequency regens, and the stability is worth the overhead.
Simultaneously, we had to stop the canonical_blog pipeline from simply freezing at the QA stage. We found that when critic model settings were empty, _resolve_critic_model would raise a RuntimeError that propagated all the way up to _wrap_atom, marking every task as halted=True (PR #1931). We wrapped that fallback call in a try/except block so it degrades to a graceful skip instead of a total halt, and we seeded pipeline_critic_model=ollama/phi4:14b into the defaults so fresh installs don't start in a broken state.
Even after fixing the settings, the pipeline still struggled because the Prefect subprocess doesn't run the FastAPI lifespan where SettingsService lives (PR #1932). We had to add a fallback path to resolve the critic model via SiteConfig when self.settings is None (af7e09a). It was a classic case of "it works in the API, but not in the worker."
On the observability side, we caught a routing bug in brain/alert_sync (PR #1934). We had hardcoded datasourceUid: "prometheus" for every rule, meaning our SQL-driven alerts were being sent to Prometheus--which obviously can't execute SQL. This caused every 60s eval cycle to fail with "data source not found" (02e5355). By adding datasource_type to _hash_rule, we invalidated the stale hashes and let the brain sync cycle auto-recover the routing to local-brain-db.
We cleaned up a few more regressions before cutting release 0.87.1 (PR #1937):
- Fixed a frontend bug where missing posts were returning HTTP 200 instead of 404 because
generateMetadatawas committing the status too early (PR #1925). - Registered
embeddings_collapseandembeddings_orphan_pruneinload_all()after realizing they'd been left off the explicit import list inhandlers/__init__.py(PR #1933).
These fixes don't add new features, but they close the gap between "it works on my machine" and a resilient autonomous system. We're finally moving past the fragility of the QA pipeline; now we can actually trust the critic to do its job.
Auto-compiled by Poindexter from today's commits and PRs. See the work: github.com/Glad-Labs/poindexter.
Top comments (0)