DEV Community: Matthew Gladding

The Question We Actually Had to Answer

Matthew Gladding — Fri, 17 Jul 2026 18:33:53 +0000

Every content pipeline eventually hits the same moment: something is rejecting good work, or something is letting garbage through, and you don't know which. We hit it hard. Our QA rail was killing roughly six out of every ten pieces that made it that far, and the easy story -- "the critic is broken, loosen the gate" -- was sitting right there, ready to be believed.

We didn't believe it. We audited instead.

The Baseline

Here's what the pipeline looks like day to day: a writer model (gemma-4-31B) drafts, a critic model (phi4:14b) reviews, and anything scoring under our qa_final_score_threshold of 80 gets one shot at a rewrite before it's killed for good. Over a recent 30-day window, we published 41 pieces at an average version score of 78.5, with 4 sitting in an approval queue. In the same window, 72 pieces got rejected -- 57 outright, 15 after already failing a rewrite attempt. That's a roughly 62% rejection rate on anything that reached QA at all.

That number alone doesn't tell you if the critic is doing its job. It tells you the critic is doing something. The question was whether that something was catching real defects or just being trigger-happy.

Running the Audit

So we pulled all 72 rejects and went through them one at a time, attributing each to whichever gate actually killed it. The breakdown:

Cause	Count
Critic hard-FAIL (only gate that failed)	20
Programmatic validator FAIL	29
Brand-fabrication FAIL	10
No hard fail at all	remainder

The critic alone only accounted for 20 of the 72. Most of the carnage was coming from the programmatic validator and brand-fabrication checks -- mechanical gates, not the model doing judgment calls.

Where the Rejections Actually Came From

Digging into the programmatic validator failures turned up something upstream of QA entirely: reasoning tokens from models like glm-4.7-5090 and gemma-4:31b were leaking into finished output, corrupting titles and getting flagged as broken structure -- not bad writing. We fixed that at the source with a strip_reasoning_artifacts function rather than telling the critic to look the other way. That's the difference between patching a symptom and fixing the actual defect.

Once we isolated those cases, the picture on the critic's own 20 hard-fails held up. We went through each one manually. The verdict: the critic is exonerated. Phi4 was catching real defects -- thin content, broken claims, structural problems -- almost every time. Our false-positive hypothesis was wrong.

The Critic Is Doing Its Job -- the Problem Sits Upstream

This matters more than it sounds like it should, because the instinct when a gate is expensive is to weaken the gate. That instinct is almost always backwards. If a critic model is rejecting 60% of what it sees and the rejections are justified, loosening the threshold doesn't fix your pipeline -- it just moves the mess further downstream, onto readers.

Fresh-eyes review matters here too. Ours runs as a separate model from the writer specifically so it isn't grading its own homework, the same principle we've written about in how AI amplifies existing engineering practices -- a QA layer that shares blind spots with the thing it's checking isn't a QA layer, it's theater.

The lesson from this audit wasn't "add more automation." It was narrower: audit before you tune. A critic that looks broken because your reject rate is high might just be honest. Find the actual upstream cause -- in our case, leaking reasoning tokens -- before you touch the one gate that's doing exactly what you built it to do.

Hunting ghosts in the metrics and tightening the CI ratchet

Matthew Gladding — Fri, 17 Jul 2026 13:03:55 +0000

What we shipped on 2026-07-17

We spent today cleaning up "ghost" bugs--the kind of silent failures that don't trigger alerts but hollow out your data. The most jarring was discovering that lab and writer observability had been dead since June (PR #2649). We found 6,243 rows in atom_runs.metrics that were completely devoid of content_length, model_used, or prompt_template_key. The culprit was a shim in services/atom_registry.py passing record_sink=None to make_stage_node, which effectively threw away all computed metrics before they could be persisted (PR #2649).

This pattern of silent failure extended into our LLM provider. While verifying other changes, we realized LiteLLMProvider.embed() was never passing the api_base to litellm.aembedding (PR #2657). Because it fell back to a default Ollama endpoint at localhost:11434, every single embed call failed inside the worker container while the logs stayed deceptively quiet (PR #2657). We also caught our QA rails lying to us; self_consistency_rail.evaluate() was returning a score of 1.0 even on skip paths, meaning "didn't check" and "perfect" were indistinguishable in the audit log (PR #2655).

On the security front, we finally stopped the Bandit linter from acting as a GitHub-issue firehose (PR #2650). After triaging 91 issues and finding almost all were false positives--mostly because Bandit lacks dataflow analysis for our services/ patterns--we moved to a ratchet system. We grandfathered existing findings into scripts/ci/bandit_baseline.json, so CI now only blocks net-new findings (PR #2650). We tightened that baseline from 61 down to 38 findings after burning down the B608s (PR #2651) and pinned HF model revisions across our image and video servers using from_pretrained and hf_hub_download to close a supply-chain exposure (PR #2658).

We also dealt with some regressions in the media pipeline. A previous rename of sdxl to image_gen had been shipped without a backcompat shim, causing older frozen shot lists to fail validation (PR #2652). We found one production task that had burned through 304 checkpoints--re-paying for TTS and transcription over and over--because the media_reconciliation watchdog kept re-dispatching it into a validation failure (PR #2652). We also patched a latent defect in podcast_persist.run where an OSError during a file move returned an empty list instead of an error, masking disk failures as legitimate no-ops (PR #2656).

It was a day of hardening. By turning the linter into a ratchet and surfacing these silent failures, we're moving away from hoping the system is healthy toward actually proving it.

Auto-compiled by Poindexter from today's commits and PRs. See the work: github.com/Glad-Labs/poindexter.

Sources

https://github.com/Glad-Labs/poindexter

Naming Lies and Frozen Tails

Matthew Gladding — Thu, 16 Jul 2026 13:04:00 +0000

What we shipped on 2026-07-16

We caught a "Sonnet-canary leak" today that serves as a stark reminder of how dangerous misnomers are in an LLM pipeline. We had pinned pipeline_writer_model to a paid model for a blog-writer experiment, but our satellite phases--things like self_consistency_rail and narrate_bundle--were silently billing at cloud rates (PR #2634). The culprit was a function called resolve_local_model which, despite its name, returned the writer pin verbatim. We've since renamed it to resolve_writer_model to stop lying to ourselves (PR #2638) and migrated those satellite phases over to a new, guaranteed-local resolver: resolve_local_writer_model (PR #2636).

The video renderer also needed some discipline. A 30-day audit of our canonical blog videos revealed that long-form renders were averaging ~171s against a 253s plan because the system was silently dropping any shot that failed to render (PR #2633). We implemented a "never-drop-a-shot fallback ladder" via _backfill_pass in shot_list_renderer.py. Now, if a primary source fails, it attempts a cross-family substitute before finally falling back to a guaranteed branded card--ensuring the video length actually matches the plan.

We also killed the "frozen tail" bug affecting our 9:16 short-form videos (PR #2637). The visuals and narration were being sized independently, leading to cases where the script ran long and the compositor simply cloned the final frame to fill the gap. We fixed this by introducing narration_fit in render_shot_list, which uses _fit_scene_durations to proportionally rescale scenes to match the real TTS duration.

On the ops side, we closed out the follow-ups from our B2 storage-cap incident (PR #2635). We learned the hard way that restic forget --prune doesn't actually free billed capacity on B2 without a version-expiry lifecycle rule. We've updated the reclaim runbook and improved our alerting so that offsite_backup_failed now includes the actual restic stderr instead of just a return code.

Finally, we added deep observability to the two-pass writer path (atoms.two_pass_writer) (PR #2639). We're now capturing per-call prompt-size metrics broken down by context section--RAG snippets, research, dev_diary bundles, and internal-grounding anchors--surfacing them as a new "Writer Context Size" row on the Pipeline dashboard.

We're finally seeing the gap close between what the architect plans and what actually renders. From here, we can start tuning the prompt sizes now that we actually have the telemetry to see where the bloat is.

Auto-compiled by Poindexter from today's commits and PRs. See the work: github.com/Glad-Labs/poindexter.

Sources

https://github.com/Glad-Labs/poindexter

The Bug That Was Already Fixed -- Five Weeks Ago, By Code Nobody Was Calling

Matthew Gladding — Wed, 15 Jul 2026 13:03:46 +0000

What we shipped on 2026-07-15

Today's most interesting fix wasn't a fix at all -- it was an autopsy. The issue (poindexter#757) proposed adding a lease_expires_at column plus a heartbeat to distinguish a stalled worker from a dead one. Investigation found we didn't need any of it: TasksDatabase.heartbeat_task already existed, built specifically to refresh pipeline_tasks.updated_at so sweep_stale_tasks could tell "actively processing" from "worker died mid-stage." It just had no caller. Its only caller, TaskExecutor, was deleted whole in the 2026-05-16 Prefect cutover when content_generation.py took over, and nobody re-wired the heartbeat into the new flow. Every stale-sweep decision since has been judging purely by claim-time age (PR #2528). That's the kind of bug that doesn't throw -- it just quietly makes the wrong call for weeks, and the fix was re-wiring a call site, not inventing a schema.

The social pipeline had its own version of "no error, wrong outcome." social.generate_drafts was catching generation exceptions with a log-only except -- no re-raise, no operator alert, no persisted failure row -- so a transient LLM error left zero social_post_drafts rows while atom_runs still reported status='ok'. RetryFailedSocialDraftsJob couldn't touch it because there was never a failed row to retry. We confirmed 2 of the last 6 published canonical_blog posts had silently lost their social drafts this way. Both text-platform and per-subreddit paths now call notify_operator -- batched once per run for Reddit rather than once per subreddit -- and we added BackfillMissingSocialDraftsJob plus a poindexter social backfill CLI command to reconcile published posts against their template's graph_def, safe to run repeatedly since the atom's own existing_draft_keys() idempotency guard makes a fully-covered post a cheap no-op (PR #2532).

Test suite got its own ghost-in-the-machine treatment: nest_asyncio.apply() permanently monkeypatches asyncio.run process-wide with no unapply, so a single test faithfully reproducing Ragas's reentrant-loop bridge poisoned every later asyncio.run() call in the same pytest process -- breaking a revalidation test's "two calls, two distinct event loops" assumption, but only when run as part of the full suite, never in isolation. Fix was _isolate_nest_asyncio_patch(), a snapshot/restore context manager scoped to the offending test (PR #2529). Findings dashboard had a smaller but similarly sneaky split: emit_finding()'s contract says severity "warn", but nine call sites had drifted to "warning" over time, fragmenting one severity tier into two badges -- WARN·1172 next to WARNING·38 -- even though the router already treated them as equivalent for routing. Normalized at the write boundary now; live-data check confirmed 1172+38=1210 raw merges to 1211 after the one more that landed mid-check (PR #2526).

Smaller stuff rounded out the day: get_logger() finally got an explicit StructuredLogger Protocol return type instead of leaving Pyright to infer Any | Logger, clearing 77 false-positive reportCallIssue errors across five files (PR #2527); the validation-failures endpoint got offset pagination (PR #2522); and we pulled _llm_semantic_pexels_query out of ImageService into build_semantic_pexels_query(topic, *, site_config) -- a pure function masquerading as a method, the first real cut against a file that's grown to 1,345 lines despite three rounds of "we should shrink this" (PR #2524). All of it rolled into 0.104.0 (PR #2625), alongside a new ProbeDisabledCapabilitiesJob for surfacing silent opt-in features and per-niche cadence targets wired into the SLO probe.

None of these were glamorous fixes. The heartbeat one is the one we'll remember -- proof that sometimes the right fix isn't the one the issue asked for, it's finding the mechanism that was already built and just never got its wire connected.

Auto-compiled by Poindexter from today's commits and PRs. See the work: github.com/Glad-Labs/poindexter.

Sources

https://github.com/Glad-Labs/poindexter

Where the Silent Failures Were Hiding

Matthew Gladding — Wed, 15 Jul 2026 00:04:00 +0000

What we shipped on 2026-07-14

Today's release notes read like a night watchman's log -- five different flavors of "nobody noticed this was broken," found and closed in the same sitting.

The most alarming one: qa.vision's dedicated Ollama instance, poindexter-ollama-vision-gpu1 on port 11435, had been dead for over a day. GlitchTip issue #877 was climbing -- APIConnectionError: OllamaException counts ticking 95→105 and up -- and the box hadn't crashed, it just quietly stopped answering. Root cause was almost funny once we found it: the scheduled task from background-services.ps1 only triggers AtLogOn, and ollama.exe serve had exited clean (code 0) during a host reboot on 2026-07-13. Windows' RestartCount/RestartInterval only re-arms on a nonzero exit, so a graceful death was invisible to its own safety net. vision_gate is required_to_pass=false, so no posts got blocked -- but the final visual reviewer had been silently absent the whole time, and every pipeline run was logging one more Sentry exception into the noise. Install-Services now re-triggers on the repeating schedule instead of trusting logon events (PR #2500).

That same "silent pass that isn't a pass" shape showed up in the QA gates themselves. multi_model_qa.py::_run_gate_prompt had two branches that swallowed real failures: a completion that comes back non-None but with empty text, and a regex-extracted JSON candidate that still fails to parse on a second attempt -- both returned None without a peep, indistinguishable from a gate that just decided to skip. The issue's framing stuck with us: "a crashed gate looks like a disabled/passing gate." Behavior's unchanged, gates still skip on failure -- but now both paths log a warning, closing the last two sites from the original no-silent-failures sweep (PR #2504).

The nicest bug of the day was buried in markdown-fence stripping. Three different places in the codebase reimplemented "strip the

json fence off this string," and one of them -- `generate_video_shot_list.py::_extract_json_object` -- used a non-greedy regex that matches the *shortest* span to the next triple-backtick. Feed it a director response containing an embedded code example like `wrap code like` ``

this


 `for emphasis`, and the parser truncates right there, silently dropping the entire shot list. Genuinely latent, not just style drift -- we reproduced it with a regression test before consolidating all three call sites onto one canonical `strip_markdown_fence` in `services/llm_text.py` (PR #2499).

Two more from the same family of "the code and the docs disagreed and nobody had checked": `writer_core.py` was calling `self_review_and_revise` inline *and* letting the dedicated `writer_self_review` graph node run it again -- for anyone with `enable_writer_self_review=true`, every draft got reviewed twice for zero benefit, just double the latency and cost. And `rag_hybrid_enabled` had been documented as live-on-prod in CLAUDE.md while defaulting `false` in code -- pure SQL (BM25 tsvector plus pgvector RRF fusion), no reason for the drift, so we flipped the default to match reality (PR #2507).

On the security side: `gsc_main`/`ga4_main`'s Google OAuth `client_secret` and `refresh_token` had been sitting in plaintext inside `external_taps.config`'s unencrypted jsonb column -- readable by any SQL session or DB dump. `app_settings.is_secret=true` plus `SiteConfig.get_secret()` is already the established pattern everywhere else; this closed the one place that hadn't adopted it. `tap.singer_subprocess` now takes a `secret_fields` map, resolves each key immediately before the subprocess spawns, and merges it into a copy so the resolved value never touches `row.config`. A forward-only migration salvages any still-plaintext secret into `app_settings` first, then strips it (PR #2502).

Smaller wins rounded out the release: `list_categories`/`list_tags` were fetching every row and slicing in Python -- now they push `LIMIT`/`OFFSET` into SQL with `COUNT(*) OVER ()`, matching the pattern `list_posts` already used (PR #2501). `probe_traffic_anomaly` was comparing partial-day views since midnight UTC against a full 24h average -- for the first nine or ten hours of any day that reads as a 60-100% traffic "drop" no matter what's actually happening -- now it compares rolling trailing-24h windows instead (PR #2498). And on the feature side, `tasks add-image`/`remove-image` finally close the CLI's image-editing gap: no more pasting raw `<img>` tags to remove an inline image, no more needing a URL just to null the featured slot (PR #2503).

None of these were dramatic on their own. Together they're the theme of the day: several places where the system was quietly wrong and nothing was loud enough to tell us. 0.103.0 (PR #2511) ships all of it at once -- the value wasn't in any single fix, it was in going looking for the other four.

_Auto-compiled by Poindexter from today's commits and PRs. [See the work: github.com/Glad-Labs/poindexter](https://github.com/Glad-Labs/poindexter)._

## Sources
- <https://github.com/Glad-Labs/poindexter>

The Console Was Lying to Us About Blocked Approvals

Matthew Gladding — Mon, 13 Jul 2026 13:03:43 +0000

What we shipped on 2026-07-13

Today's most useful fix wasn't a feature -- it was admitting the console had been quietly lying. POST /api/social/drafts/{id}/approve returned HTTP 200 even when SocialDraftsService.approve_draft() blocked the push, and the console's fetch wrapper only checks status codes. So a blocked approval -- promoted post not yet published, missing Postiz integration UUID -- painted a green "queued for Postiz" toast and the draft vanished from the queue while nothing actually reached Postiz. We confirmed it against prod data: two drafts for a specific task had sat pending since 2026-07-12 because the task was still awaiting_approval. Every approve click on those was silently swallowed. The fix is a contract match -- routes/social_routes.py now raises HTTPException(409, detail=error), matching the wrong-state-409 pattern approval_routes.py already established, and console/js/api.js's http() now folds the response body's detail/error into the thrown error so toasts show the real reason (PR #2435). CLI and MCP never had this problem -- they read the result dict directly -- which is its own small lesson about how many failure modes only exist at the HTTP boundary.

The other satisfying arc today was watching metrics go from collected-but-invisible to actually on a dashboard. plugins/scheduler.py::_runner had been discarding JobResult.metrics for every job -- 44 jobs already populate it, logging only to Loki -- while the dashboards saw nothing. _capture_job_metrics now writes an event_type='job_run' audit_log row per metrics-emitting fire, namespaced so job keys can't shadow the envelope (PR #2431). That data sat queryable but panel-less until the Job Fleet Metrics row went up on Integrations & Admin -- job-agnostic, working immediately for media_reconciliation, render_*, sync_cloudflare_analytics and friends without per-job wiring (PR #2436). Then we went further and cross-referenced every poindexter_* series against every provisioned dashboard's panels[], finding metrics that were live in Prometheus and displayed nowhere -- 13 new panels across 7 dashboards, all exprs validated against the live instance (PR #2442). Three PRs closing the same gap from different angles: emit, surface, audit for completeness.

We also shipped the foundation piece Dev.to syndication actually needed. The existing niche taxonomy couldn't separate ai-ml from gaming/hardware -- everything's glad-labs, and tags are LLM-sprawl -- so "syndicate ai-ml, exclude gaming" wasn't expressible as a rule. The new post_content_types table gives us a multi-label axis, populated by ClassifyContentTypesJob sweeping published posts every 6h through a local structured-extraction LLM against a DB-configurable label set. The guardrail we care about: only labels in the set persist, and a post the model can't confidently classify gets zero rows rather than a silent default. With that axis in place, CrosspostToDevtoJob finally gates on an operator allowlist plus a quality floor instead of syndicating every published post indiscriminately (PR #2415).

Smaller but no less real: we finally nailed down why Z-Image-Turbo keeps rendering mangled text into images. It's guidance-distilled (guidance_scale=0), which means its own image_negative_prompt -- the one that lists "text, words, letters, numbers, watermark" -- never applies, and diffusers' ZImagePipeline ignores negative_prompt outright when guidance_scale < 1. The "no text" lever isn't tuned wrong, it's structurally dead on this model. That sent us into a bake-off spec across twelve candidate text-to-image models, each scored at its own best config rather than forcing one knob onto all of them (PR #2386).

Release 0.102.0 bundled all of it plus the smaller connective tissue -- the Grafana alert sync CLI, affiliate multi-keyword matching with CSV bulk-import, footer icons for YouTube and Buy Me a Coffee, the nightly app-settings.md and CLAUDE.md doc-sync bots keeping source-of-truth stats honest without a human in the loop (PR #2445, PR #2444, PR #2443). None of it is glamorous individually. Together it's a day where the dashboards stopped lying about coverage and the console stopped lying about success -- which, on a one-person shop, is worth more than any single feature.

Auto-compiled by Poindexter from today's commits and PRs. See the work: github.com/Glad-Labs/poindexter.

Sources

https://github.com/Glad-Labs/poindexter

The Echo Chamber Problem, From the Inside

Matthew Gladding — Mon, 13 Jul 2026 02:31:12 +0000

We didn't set out to write about echo chambers. We set out to fix two content tasks that were producing off-topic garbage. What we found underneath was a textbook case of a retrieval system reinforcing its own mistakes.

The pollution we traced

A task titled "Knowledge Distillation of Black-Box Large Language Models (2024)" came back with content that had nothing to do with the topic. Same story on a second task. When we pulled the content_revisions rows, the initial_draft content didn't match the assigned topic at all.

The root cause was retrieval, not generation. Our writer's RAG layer was supposed to pull only from our own published content -- a rag_source_filter restricting retrieval to content-only sources. Somewhere along the way, that filter wasn't consistently applied, and the writer was retrieving snippets that had no business grounding that topic. We fixed it by hard-coding the constraint into the retrieval layer itself: writer RAG now runs content-only, always, with the two-pass snippet query wired through every entry point so nothing slips through unfiltered again.

Two investments, two code paths

The deeper audit was more interesting than the bug fix. We'd made two real investments in retrieval quality over time: a hybrid-plus-rerank pipeline for better relevance, and a Maximum Marginal Relevance pass to cut down on retrieved-document redundancy -- the same problem Nick Berens describes as the "similarity trap," where a RAG system returns three near-identical documents instead of one useful one plus something new.

Here's the catch: those two systems live on different code paths in our stack. Neither path has both. The writer's primary grounding query -- the one doing the actual heavy lifting on every article -- runs through the least-capable retriever we have. It gets neither the rerank quality boost nor the diversity fix. The fancy stuff exists. It's just not attached to the thing that matters most.

That's the echo chamber in miniature: not a philosophical problem about AI sycophancy, but a plumbing problem where your best retrieval logic and your production query never actually meet.

The de-echo layer we built

We also found a second, related failure mode: the writer's draft would sometimes echo its own prompt back -- restating the topic, angle, and instructions almost verbatim in the first few lines, instead of writing content. We built a detector for this with two signatures: identity echo (the topic, angle, or override restated) and instruction echo (imperative lines from the prompt showing up in the draft). A structural detector we added afterward catches the more blatant version -- a draft that's mostly bullet points dumped from the planning stage rather than prose.

None of this is unique to us. Projects like Xtra-Large-Bubble are explicitly testing for bias loops in RAG-generated answers, and the discussion on Scrapbox asks the same question we ran into by accident: does retrieval make a model more likely to just repeat convenient information back at you, rather than actually reasoning about it.

Where this leaves us

The fix isn't glamorous. It's making sure the retriever with rerank and MMR actually serves the writer's primary query, and making sure a content filter that's supposed to be always-on is actually always on. If you're running a RAG pipeline of your own -- whether it's built on LangChain or Hugging Face's RAG stack -- check which retriever your production path actually hits. The best components in your repo don't help you if they're sitting on a branch nobody calls.

Sources

Why a 2021 Textbook on Compilers Still Matters

Matthew Gladding — Sun, 12 Jul 2026 22:31:12 +0000

Compiler theory doesn't have a "move fast and break things" phase. The fundamentals -- lexing, parsing, semantic analysis, code generation -- have been settled for decades. What changes is who's reading the textbooks.

In 2021, Douglas Thain, a computer science professor at the University of Notre Dame, released a refreshed edition of Introduction to Compilers and Language Design, built around his CSE 40243 compilers course. It's not a niche release. It's become one of the more widely mirrored free compiler textbooks online, showing up as a standalone PDF, on library aggregators like FreeComputerBooks, and as a paperback and hardcover on Amazon.

That spread matters for indie developers and tinkerers specifically. Most of us never took a formal compilers course. We learned Python or C# or Lua as consumers of a language, not as people who understood how the interpreter under the hood was actually built. A free, well-structured textbook lowers the barrier to fixing that gap -- and if you're building anything with a custom scripting layer (a modding API, a shader DSL, a save-file format with logic embedded in it), that gap is exactly where bugs live.

This post walks through what the book covers, how it fits into the broader landscape of compiler education around 2021, and what we've actually learned building our own interpreter here at Glad Labs -- where the textbook's abstractions meet the messy reality of a working codebase.

What a Compiler Actually Does

Before getting into the book itself, it's worth being precise about the term. According to Wikipedia's entry on compilers, a compiler is software that translates code written in one programming language into another -- typically from a high-level source language down toward something a machine (or virtual machine) can execute directly.

The pipeline that Wikipedia lays out is the same one every compilers course teaches: source code goes in, gets turned into an intermediate representation, and eventually comes out as object code, bytecode, or machine code. Along the way you've got compile time versus runtime, linking, and execution -- concepts that show up whether you're writing a C compiler targeting x86 or a toy interpreter for a game's scripting language.

The distinction that trips people up early is compiler versus interpreter versus virtual machine. A compiler translates ahead of time. An interpreter executes source (or an intermediate form) directly, on the fly. A virtual machine sits in between -- it executes a compiled intermediate representation, but does so at runtime, which is how a JVM or a CLR-based .NET runtime behaves. Thain's book, like most compilers courses, treats these as points on a spectrum rather than separate disciplines, because the front end -- lexing and parsing -- looks nearly identical regardless of what you do with the result afterward.

What Thain's Book Actually Covers

The book opens with the basics: what a compiler is, why you'd study one, and how the traditional academic answer ("optimizing performance for a fixed target machine") coexists today with a broader practical answer -- building interpreters, transpilers, and domain-specific languages for problems that have nothing to do with CPU architecture.

The publisher listing on FreeComputerBooks confirms the book was independently published in June 2020, with a free digital edition released in 2023, and it's explicitly framed as a one-semester course -- the same pacing Thain uses for his Notre Dame class. That's a deliberate design choice. A lot of compiler textbooks are written for a two-semester graduate sequence and assume you'll spend a full year on it. Thain's version is scoped so a working developer, not just a full-time student, can get through it.

A companion summary of the 2021 edition on FATSIL frames the book as targeting "students and developers" specifically -- not just an academic audience -- with the stated aim of improving foundational understanding of programming language development. A separate write-up from RottenPanda describes the update as focused on modern language features and compiler techniques, positioning it as a comprehensive guide rather than a narrow reference.

The practical upshot: this isn't a book you read cover to cover and then set aside. It's structured so you build a working compiler alongside it, chapter by chapter, the same way Thain's own students do over a semester. Each chapter tends to hand you a working piece of the pipeline before asking you to extend it, which is a deliberately different rhythm than a reference text you'd dip into for a single answer.

How It Stacks Up Against a University Course

It's worth comparing Thain's self-contained textbook to what an actual university compilers course looked like in the same period. Cornell's course, which covers the specification and implementation of modern compilers, is a useful artifact in its own right, because the course page is frozen at a moment when in-person and remote instruction were both live options, mid-pandemic.

The overlap between a course like Cornell's and Thain's textbook isn't a coincidence. Both are teaching the same core sequence: lexical analysis, parsing, type checking, intermediate representation design, and code generation. What differs is delivery. A university course comes with assignments, a grader, and office hours. A free textbook comes with none of that -- which means the responsibility for building something and testing it against reality falls entirely on you.

That's not a knock against self-study. It's just a reason to treat the book as a lab manual, not a novel. If you're going through Thain's material without a professor checking your work, the only way you'll know your lexer or parser is actually correct is by running it against real input and watching it fail in interesting ways -- feeding it malformed programs on purpose, checking that error messages point at the right line, and confirming that edge cases like empty input or deeply nested expressions don't quietly break your assumptions.

Where Language Design Fits Alongside Compiler Construction

Compiler construction and language design are related but distinct disciplines, and it's worth separating them clearly, because a lot of the pain in building your own language comes from conflating the two.

Compiler construction is mechanical: given a language specification, how do you build software that correctly implements it? Language design is the harder, less mechanical question: what should the language actually look like, and why?

A developer blog from Sofia Celi makes this distinction well, tracing her own interest in language design back to studying Go's compiler and design decisions. She points to two influential papers behind Go's design choices: C.A.R. Hoare's "Hints on Programming Language Design," and a related paper on language design tradeoffs. The throughline in her post is that the decisions baked into a language's syntax and semantics -- not just how fast the compiler runs -- determine whether a language is pleasant or miserable to actually use, and that this is a judgment call language designers make long before a single line of the compiler gets written.

That distinction matters if you're an indie developer thinking about building a scripting layer for a game, a modding API, or a config format with embedded logic. The compiler construction side (Thain's book, the Cornell course, the Wikipedia pipeline) tells you how to build the machinery. The language design side tells you what decisions you're actually making when you choose, say, whether your scripting language has implicit type coercion, whether it's whitespace-sensitive, or whether it exposes mutable global state to modders. Get the mechanics right and design the language poorly, and you'll ship something technically correct that nobody enjoys writing scripts in.

What This Looks Like in Practice: Our Own Interpreter

We've been building our own interpreter in Python as part of our internal tooling, and it's been a direct, hands-on encounter with exactly the problems these textbooks describe in the abstract.

The most significant challenge we've run into is the classic bootstrapping problem: circular dependency. The interpreter needs to be able to parse and execute code, but pieces of the interpreter's own logic end up depending on structures that are only fully defined once the interpreter itself is running. This is the same self-hosting tension compiler writers have dealt with for decades -- a compiler that can compile itself, an interpreter that can interpret its own control flow -- except we hit it as a live engineering problem, not a textbook exercise.

Textbooks like Thain's tend to present the compiler pipeline as a clean, linear sequence: lexer, then parser, then semantic analysis, then codegen or execution. In practice, when you're writing the thing yourself in a dynamic language like Python, that linearity breaks down the moment you try to make any part of the system reusable or introspectable. You end up needing to think carefully about initialization order, about what state exists before the interpreter can execute anything at all, and about how much of the "chicken and egg" problem you're willing to solve versus work around. We've landed, more than once, on deliberately deferring the construction of certain runtime structures until the first pass of parsing is complete, rather than trying to make everything available up front -- a workaround that no textbook diagram quite prepares you for.

This is the value of a book like Thain's for someone doing this kind of work: it gives you vocabulary and a mental model for the pipeline before you're in the trenches fighting circular imports. Reading the theory first doesn't make the engineering easier, but it does mean you recognize the shape of the problem when you hit it, instead of debugging it as if it were a one-off bug in your specific codebase.

Why Indie Developers and Tinkerers Should Care

Glad Labs covers AI/ML, gaming, and PC hardware, and the compilers-and-language-design intersection touches all three, just in different ways.

On the gaming side, most engines expose some kind of embedded scripting language or visual scripting system, and plenty of successful indie titles have shipped their own tiny DSLs for dialogue trees, quest logic, or modding support. Understanding the compiler pipeline -- even at the level Thain's one-semester book covers -- is the difference between bolting together a scripting system that barely works and one that's maintainable, debuggable, and extendable by modders down the line.

On the AI/ML side, the connection is less obvious but still real. Anyone who's worked with prompt templating languages, model configuration DSLs, or the intermediate representations used by ML compiler stacks (think ONNX or the graph-level IRs frameworks compile down to) is working with the exact same lexer/parser/IR/codegen pipeline, just applied to a different kind of "program." We've written before about running language models with unusual training constraints -- our post on running a 13B model trained only on pre-1931 text touches on how much the shape of a model's inputs determines its behavior, and that same principle -- the structure of your input format shapes everything downstream -- is exactly what a compiler's front end is designed to handle cleanly.

On the PC hardware side, anyone tinkering with firmware, shaders, or low-level tooling runs into compiler concepts directly: cross-compilation targets, intermediate representations for GPU shader languages, and the tradeoffs between interpreted and compiled execution paths on constrained hardware.

None of this requires you to become a compiler engineer full-time. It requires enough grounding to recognize when a problem you're facing is actually a parsing problem, or a type-checking problem, or a code-generation problem -- and to know there's decades of established technique for solving it, rather than reinventing a worse version from scratch.

Getting Started Without a Semester to Spare

If you don't have a semester to dedicate to Cornell's course structure, Thain's book is designed to be approachable without one. The free chapter PDFs are available directly from his site, and the mirrored copy on FreeComputerBooks and the standalone PDF mirror both give you the same content without needing to buy anything. If you'd rather have a physical copy to work through away from a screen, the Second Edition on Amazon is the same material in print.

The practical advice, based on what we've hit building our own interpreter: don't just read the chapters in sequence and move on. Build something small alongside each section -- even a toy calculator language with a handful of operators is enough to force you to actually implement a lexer and a recursive-descent parser rather than just recognizing the terms when you see them again. The moment you try to add a feature that touches more than one part of the pipeline -- say, adding variable scoping to your toy language -- you'll start running into the same structural problems real compiler writers deal with, just at a smaller scale.

And if you get to the point where you're building something that needs to be self-hosting, or where parts of your interpreter's logic need to reference structures defined by the interpreter itself, expect it to be harder than the book makes it look. That's not a flaw in the material. It's the gap between describing a pipeline and actually building one that survives contact with a real codebase.

Closing Thoughts

A free, well-scoped compiler textbook showing up in 2021 and staying relevant since is a genuinely useful thing for the indie dev and tinkerer crowd Glad Labs writes for. It gives you the vocabulary and mental model -- lexer, parser, IR, codegen -- that turns "my scripting language is buggy" into a specific, solvable engineering problem instead of a vague sense of dread.

Thain's book, alongside course material like Cornell's compilers sequence and language-design writing like Sofia Celi's post on Go's design influences, covers both halves of the problem: how to build the machinery, and how to decide what the machinery should actually do. Our own experience building an interpreter has made clear that the textbook version and the practical version rhyme but don't match exactly -- the circular dependency problems and bootstrapping headaches show up regardless of how clean the theory looks on paper. That's exactly why it's worth reading the theory first: you'll recognize the problem faster when you're the one debugging it at 11pm.

Sources

The Day We Spent Correcting Ourselves

Matthew Gladding — Sun, 12 Jul 2026 13:03:27 +0000

What we shipped on 2026-07-12

Today's most instructive fix wasn't a fix at all -- it was admitting the last fix was wrong. fix(power): stop false PSU-floor alert storm (PR #2371) went after a real problem: the brain paging Telegram roughly 15 times a day with "No real PSU data," every page false, the Shelly plug reading a steady ~330W the whole time. Root cause: the host exporter built /metrics synchronously per request -- nvidia-smi + AIDA + Shelly poll, measured at 2.5-6.7s, sometimes wedging Prometheus to 275s -- against a 3s scrape timeout. Eighteen floor-cycles per 24h, each one a page. The fix moved collection off the request path onto a background thread.

Then the fix broke something else. fix(power): don't let the nvidia-smi watchdog kill the exporter (PR #2374) landed as a same-day follow-up: moving collection to a background thread let the old #319 nvidia-smi watchdog's SystemExit escalate via os._exit, turning a previously-harmless watchdog fatal. Three consecutive nvidia-smi timeouts under GPU load tripped it, the exporter hard-exited, the host task didn't reliably restart it, and all host metrics went dark -- caught in post-deploy verification about five minutes after #2371 shipped. The watchdog existed to kill a process wedged on the request path; #2371 removed that request path, so the watchdog had gone from "necessary" to "pure downside" without anyone deleting it. The actual fix was small: the collector loop now swallows the timeout instead of escalating it.

The same pattern showed up in QA. fix(qa): qa.vision fail-open stops mispaging a healthy vision model as unavailable (PR #2364) diagnosed a vision_scorer_unavailable false-page as a None database_service.pool and shipped a site_config._pool fallback. The follow-up, fix(qa): vision false-page is a thinking-budget truncation, not a dead pool (PR #2369), found the pool was never None -- task b740e4b8 showed qwen3-vl:30b acquiring the GPU, running 11.2s and 26.4s legs, logging thousands of tokens, releasing cleanly, and then paging "unavailable." If the pool had actually been None, dispatch never happens -- no GPU, no tokens. The page was a truncated-response artifact, not infrastructure at all. A third investigation, ci(rerank): guard the real sentence-transformers import against version-skew re-locks (PR #2370), started from a report that a huggingface-hub 1.16.1 lock silently degraded the reranker in prod -- reproduction disproved it outright, confirmed live in both poindexter-worker and poindexter-prefect-worker, and no dependency change shipped.

Three separate threads, same shape: an alarm, a plausible-but-wrong diagnosis, then the real root cause once someone actually read the logs instead of trusting the alert's framing. Not glamorous work, but it's the work that keeps the pager honest.

Underneath the corrections, real forward motion. Task b740e4b8 -- the same VRAM near-duplicate draft that triggered the vision investigation -- turned out to be a ~65-70% content re-tread of two already-published posts, and it's what motivated Content-embedding topic dedup (PR 1) + RAG-self-echo spec (PR #2359): the old word_overlap topic dedup engine is semantically blind, and the VRAM dupe scored only 0.55 on title-similarity but 0.735 on content -- a title-only gate would have burned a full ~17-minute generation before catching anything. The new ContentEmbeddingDeduplicator reuses the same pgvector find_similar_posts search the create-post guard already relies on, and content_embedding is now the default topic_dedup_engine for all installs. Its companion, feat(qa): content_originality rail -- whole-post RAG self-echo detection (PR #2375), renames qa.opening_originality to qa.content_originality and broadens the scan from opening-only to whole-post chunked, max-over-chunks -- catching the class of self-echo that a 2026-06 VRAM cluster exposed, where four posts opened near-verbatim because the two_pass writer grounds every draft on the nearest published posts. It ships advisory only; no gate-decision changes today, but it's built hard-block-ready.

Rounding out the day: fix(social): reap orphaned social drafts when content is rejected (PR #2366) stops social drafts from surviving a content rejection -- traced through the same b740e4b8 task and a second case where a post was published, promoted, then retracted two days later, leaving live promos pointing at nothing. All of it landed in chore(main): release 0.101.0 (PR #2376).

Most of today's real work was unwinding yesterday's diagnoses -- the pool wasn't None, the lock wasn't broken, the watchdog wasn't the villain it used to be. That's a slower kind of progress, but it's the kind that keeps the alerts trustworthy, which is worth more than any single feature ship.

Auto-compiled by Poindexter from today's commits and PRs. See the work: github.com/Glad-Labs/poindexter.

Sources

https://github.com/Glad-Labs/poindexter

The Cards That Spasmed on Hover, and Three Dead Atoms in the Architect's Head

Matthew Gladding — Sat, 11 Jul 2026 18:31:12 +0000

What we shipped on 2026-07-11

Today's oddest bug was cosmetic and almost embarrassing: the SYSTEM PULSE cards on the console overview jumped around under the mouse. Not a layout bug in the usual sense -- .nowrun__col had picked up a max-height ceiling in #2273 with a lone overflow-y: auto, and per the CSS spec, one non-visible overflow axis forces the other to compute to auto too. So every pulse column silently became a dual-axis scroll container. A hair of horizontal overflow thrashed the h-scrollbar against the vertical cap every time the pointer moved, and the cards spasmed. We confirmed it live by reading computed styles off the running console -- overflow-x: auto on all three columns -- then clipped the x-axis explicitly with overflow-x: hidden, the same idiom .main already uses. Console suite stayed green at 156/156, nothing touched in JS. One property, root-caused by actually reading computed styles instead of guessing (PR #2285).

The architect prompt bug was quieter but scarier. The pipeline-architect's system prompt -- the thing that grounds it when composing graph specs -- still listed atoms that don't exist anymore. We audited every atom name in the COMPOSITION HEURISTICS against the live registry (63 atoms, 15 surfaced stages) and found three stale references: aggregate_reviews should've been qa.aggregate, review_with_critic should've been qa.critic (and the old atom itself was deleted back in #2278), and stage.approval_gate doesn't surface as a stage at all -- the approval gate only exists natively. Left uncorrected, any of these could have had the architect emit a spec naming a nonexistent atom, which build_graph_from_spec rejects at compile time. We also fixed a folded-channel reference -- state.qa_reviews → state.qa_rail_reviews -- to match what qa.aggregate actually requires. Cheap fix, expensive failure mode avoided (PR #2284).

The research-dedup bug took more digging. Profiling the writer prompt for one task turned up the SOURCES block rendered twice -- roughly 9,005 characters, half of it redundant. Tracing it layer by layer: 4,199 chars came in as "caller-attached" research, another ~4,806 came from a fresh ResearchService build, and pgvector's build_rag_context contributed nothing. The root cause: the task had stalled on its first run, gotten reset to pending by the stale-reclaim sweep, then re-run -- and on the re-run, _collect_research_context read the first run's persisted research_context back in as if it were caller-attached, then appended the freshly-built context on top. We fixed the re-run path to stop treating stale persisted research as new input (PR #2286).

On the pipeline side, rebuild-images stopped blocking. It used to run the entire image pipeline inline -- minutes per draft, CLI hanging -- with a forked 233-line copy of the image logic living inside ImageRebuildService. Now it enqueues a pipeline_tasks row with template_slug=image_rebuild and returns immediately; the Prefect worker drives it through the same atoms canonical_blog runs, via a new graph template (load_draft → plan → generate → featured → gate → inject → persist → END) seeded by migration 20260711_024500. This is more infrastructure than a rebuild command probably needs, but it collapses a maintenance burden -- one atom seam instead of two parallel image pipelines (PR #2287).

Housekeeping filled out the rest of the day. ProbeZeroReaderSettingsJob had flagged 50 app_settings keys with no reads past the 30-day grace window; after grepping the whole repo for each one we retired 26 that were truly dead, kept 22 (blind spots plus intentional operator config), and left one for a human call -- 245 tests passed on the way out (PR #2281). Prettier format drift across 55 files, mostly web/public-site, and 11 ESLint warnings got cleaned up, with markdown diffs hand-reviewed to avoid prettier corrupting prose near underscore-emphasis tokens (PR #2283). Hardware sensors consolidated onto AIDA64 + iCUE after confirming the hwinfo_* exporter series was already at zero and scrape latency was erratic -- HWiNFO wasn't feeding anything real anyway (PR #2290). And social drafts finally link to the actual post instead of the bare index -- the drafts had been generated before the slug existed, so we gate on approve-time and predict the slug up front (PR #2282).

All of it landed in 0.100.0 (PR #2302) -- a version bump that's really a dozen small acts of not trusting the last assumption: not trusting that HWiNFO was still useful, not trusting that the architect's prompt matched the live registry, not trusting that a re-run task's cached research was safe to reuse. None of these fixes are individually glamorous. Together they're the difference between a system that mostly works and one we can leave alone overnight.

Auto-compiled by Poindexter from today's commits and PRs. See the work: github.com/Glad-Labs/poindexter.

Sources

https://github.com/Glad-Labs/poindexter

The Shift to a Native UI

Matthew Gladding — Sat, 11 Jul 2026 02:31:11 +0000

For most of Poindexter's life, we didn't really have a UI. We had a stack of adjacent tools: Grafana dashboards for metrics, a Discord bot and a Telegram bridge for alerts, a CLI for day-to-day operations, and a handful of MCP servers doing the actual work underneath. That's a perfectly normal way to run an AI-operated pipeline. It's also a sign that nobody has committed to a real interface yet.

We've spent the last few sprints changing that. The short version: we built a console, decided it would be our primary UI going forward, and are in the process of replacing the Grafana-in-an-iframe pattern with native panels that live inside our own application. This post is about why we made that call, how we're sequencing it, and what "native" actually means once you get past the buzzword.

The problem with dashboards bolted onto a tool

Grafana is good at Grafana. It's not good at being your product. When your primary operational surface is a dashboard embedded in a page, you inherit its rendering model, its auth quirks, its refresh cadence, and its opinions about layout -- none of which were designed around your workflow.

We noticed this the hard way. Observability, Discord/Telegram alerting, the CLI, and the MCP servers had all grown up as separate side-channels rather than parts of one coherent system. When we finally asked where UI/UX actually lived in our architecture, the honest answer was: nowhere. It was scattered across tools that happened to be adjacent to each other.

That's the same failure mode the industry has been circling for a while. The Zebkit team's writeup on custom vs. native UI components makes the point plainly: native components integrate more tightly with the platform's own rendering and interaction model, while cross-platform or embedded approaches trade that tightness away for convenience. A dashboard-in-a-frame is convenience. It's not integration.

The decision: console as primary, Grafana as fallback

Once we framed it that way, the fix wasn't "add more panels to Grafana." It was "build a real UI and demote Grafana to the tool you reach for when you need to dig deeper." We committed to that explicitly: the console becomes the thing we trust to run the business day to day, and Grafana stays available for the cases where you genuinely want its query flexibility.

That commitment changes the bar for the console. A prototype dashboard can be flaky and nobody loses sleep over it. A primary UI can't be flaky: if it's silently wrong, or silently down, that's now a real operational problem, not a cosmetic one. So we didn't treat this as a UI polish task -- we treated it as a program that needed a proper plan, not just picking a fix and shipping it.

This mirrors a broader pattern outside our own stack, too. OneNorth's writeup on the mobile platform shift points out that Apple and Google's latest platform updates aren't just feature releases -- they're pushing brands to lean on native capabilities instead of generic cross-platform shells. Same instinct, different domain: once you're serious about an interface, you stop treating the platform's native primitives as optional.

Decomposing "native everything"

"Console as primary UI, native everything" sounds like one project. It isn't. It's several independent subsystems, and trying to build them all at once is how these efforts stall out.

We broke it into a foundation phase first -- reliability and trust in the basics, things like making sure the internal http() calls actually behave predictably -- because none of the flashier work matters if the plumbing underneath is shaky. Only after that foundation held up did we move to the actual native migration: replacing embedded Grafana panels with native time-series views, GPU telemetry panels, and Postgres panels built directly into the console.

We tracked this as a formal program -- the telemetry-tab native-migration effort, sub-projects A through E -- and worked through it as a sequence rather than a big-bang rewrite. Sub-project B, the first slice of native panels, got battle-tested before we moved on to the harder pieces: time-series, GPU, and Postgres panels natively rendered instead of iframed in. That program is now complete, and the panels that used to be someone else's dashboard are ours.

This connects directly to something we wrote about previously: the shift from native to upscaled. That piece was about a different domain -- rendering -- but the underlying tension is the same one. You can lean on an intermediate layer that gets you most of the way there, or you can do the harder work of building the real thing. Upscaling gets you close to native rendering quality without the cost. Embedding Grafana gets you close to a real UI without the cost. Both are legitimate tradeoffs -- until the thing you're building becomes primary. At that point, "close to native" stops being good enough, and you make the switch.

Observability isn't a special case -- it's just another module

The architectural insight that made this tractable: observability, alerting, the CLI, and the UI aren't exceptions to how the rest of the system is organized. They split the same way everything else does. The core owns the runtime and the contract; each surface -- Discord, Telegram, CLI, console -- contributes its slice through the same registration pattern the rest of the system already uses for routes and migrations.

Once we saw it that way, "build a native UI" stopped being a UI project and became an integration project. The console isn't a separate app bolted onto Poindexter. It's another module that happens to render panels instead of processing webhooks.

That framing also explains why we're comfortable calling Grafana a fallback rather than ripping it out entirely. We evaluated replacing it wholesale and decided against a hard cutover -- Grafana's ad hoc query flexibility is still useful when you need to dig into something the native panels weren't built to show. This is the same tradeoff Zebkit's comparison draws out between native and embedded components: the tightly integrated surface wins for everyday use, but the flexible, general-purpose tool still earns its keep for the edge cases it was built for. The point isn't to eliminate Grafana. It's to stop depending on it for the interface you use every day.

Where this leaves us

The pattern here isn't unique to our stack. It's the same one that pushed React Native's declarative model into mainstream mobile development -- a shift Meta's own React Native documentation frames explicitly as a move toward native rendering and interaction rather than a purely cross-platform abstraction -- and the same one behind Apple and Google's continued push toward native platform capabilities. Cross-platform and embedded approaches earn their keep early, when speed matters more than integration. They stop earning their keep once the interface becomes something people actually depend on.

We're past that line now. The console is our primary UI, the native panels -- time-series, GPU telemetry, and Postgres -- are live, and Grafana sits one click away for when we need to dig deeper. If you're still running your operations through a dashboard embedded in someone else's tool, that's fine -- until it isn't. The moment it becomes the thing you trust to run the business, native stops being a nice-to-have. That's the line we crossed, and we're not going back to the iframe.

Sources

The Timeout That Fired in Zero Milliseconds

Matthew Gladding — Fri, 10 Jul 2026 18:31:11 +0000

What we shipped on 2026-07-10

The tell was buried in a GlitchTip stack trace: Timeout passed=90.0, time taken=0.001 seconds. A timeout that fires in a millisecond isn't a timeout -- (PR #2251) chased that lie back to litellm's own aiohttp transport, which pools keep-alive connections to Ollama. In the sparse nightly window (03:00-05:00), Ollama closes idle sockets server-side; aiohttp reuses a pooled-but-dead one, the first read fails instantly, and litellm's exception_type relabels the failure litellm.Timeout → APIConnectionError. GlitchTip issue 736 paged us 13 times over five days before we found the real fault line. The fix was a transport default -- httpx instead of aiohttp -- not a longer timeout.

That was the theme of the day: things that looked like one bug and were actually two, or a symptom standing in for the real cause. GlitchTip #863 -- WARN finding lost on audit write -- fired 92 times and turned out to have two unrelated root causes both making the audit_log INSERT raise. The louder one was a pool-close race: the Prefect content flow builds and tears down its own DB pool per run, and when spend-throttle fired mid-flight, close() beat the fire-and-forget audit write to the punch. Throttle state resets every fresh subprocess, so the same ~2-minute run kept re-losing the same finding -- hence the count. audit_log._pending_writes plus a drain_pending_writes() call before DatabaseService.close() (PR #2248) closes that race without touching the loud-drop escalation logic that surfaced it in the first place.

The social drafts atom had a quieter version of the same class of bug: state.get("pool") was reading a key that never got seeded. The canonical_blog graph path only ever sets state["database_service"], and the sibling atoms (media_persist, podcast_persist) already knew that -- this one was the odd one out, silently swallowing its own exception and quietly generating zero social drafts across fourteen posts before GlitchTip issue 855 caught it (PR #2245). The fix is the idiom the rest of the codebase already uses: getattr(database_service, "pool", None).

Image generation had its own wedged-but-healthy failure: every post from 07-08 16:09 onward shipped a Pexels stock photo instead of a generated one, both featured and inline, and it took walking posts.metadata->>'featured_image_data' timestamps to prove the last real local generation was hours earlier than the first fallback (PR #2244). Meanwhile the analytics side got its own forensic pass -- beacon raw traffic was 1,708 against GA's 178 over the same window, with one Chrome UA on Linux accounting for 1,532 hits, 90% of the noise. page_views.is_bot plus a page_views_human view (PR #2246) separates "is the pipe flowing" (raw, for liveness alerts) from "how many readers" (human, for everything reader-facing) -- a distinction we'd been fudging for a while.

Not everything today was forensic. We retired the legacy :9837 host-slideshow lane entirely -- dead weight since the shot-list renderer took over months ago, never selected by any live job, kept alive only by the recurring temptation to "fix" its host.docker.internal URL (PR #2254). And the Hardware & Power panel now meters true wall draw off a Shelly smart plug instead of fighting iCUE for the HX1500i's USB bus (PR #2249) -- get_shelly_psu_metrics() slots into the existing psu_power.py::select_power_source preference with no downstream changes, inert until the operator sets shelly_psu_url.

All of it landed in 0.99.0 (PR #2257) alongside the Trace-tab console rework and the media-QA Layer 2 semantic scoring. The through-line across today's fixes wasn't a single system -- it was pool lifecycles and stale connections quietly lying about their own failure modes. Worth remembering next time a timeout looks too clean.

Auto-compiled by Poindexter from today's commits and PRs. See the work: github.com/Glad-Labs/poindexter.

Sources

https://github.com/Glad-Labs/poindexter