DEV Community: Dan Mercede

The Recon Refuted the Task: Building Infrastructure With an Agent You Can Trust

Dan Mercede — Sun, 12 Jul 2026 04:57:27 +0000

I asked an agent to build a monitor: something to watch the background jobs across my servers and tell me when one broke. It came back and told me not to build it.

Not out of laziness. It had spent its first hour running probes against the live machines instead of writing code, and the probes said the thing I asked for already existed five times over. One platform watched the hosts. A container metrics collector watched the containers. The init system watched its own units. Two more scripts watched the edges. What I was actually missing was not another detector. It was the layer that reads five noisy signals and says "these three are the same incident, here is the likely cause, here is the runbook that fixes it."

That refusal was the single most valuable thing the agent did on the whole project. A detector I did not need would have shipped in an afternoon, run green forever, and watched nothing that mattered. The hour of recon that killed it is the reason I trust what got built instead.

This guide is the anatomy of that trust. It is worth nothing that an agent seems careful. What matters is the specific, checkable structure that let this one build a piece of production monitoring I now depend on: a recon step that argues with the task, a finish line a model can actually verify, a plan I approved before any code existed, autonomy bounded by construction rather than by a polite prompt, and acceptance by firing every control instead of watching it sit green.

Who this is for: anyone pointing a coding agent at real infrastructure, especially self-hosted systems where a quiet failure hides for weeks and "it ran without erroring" is not the same as "it works."

Why should an agent's first job be to refute your task?

In a 2026-07 build, the recon phase refuted the premise of the task before a line of code existed. I had asked for a holistic watcher because "nothing watches the jobs holistically," and a live sweep found five watchers already running. Four background units were failing at that exact moment, invisible because the watcher's coverage list was fixed at install time and never grew as new units shipped.

This is the crown rule of pointing an agent at infrastructure: recon is not information-gathering, it is error-correction. A first draft of any plan is full of confident, wrong premises, because the person writing the task does not have the live state in front of them. The job of the recon step is to take each premise and try to break it against the real system. "Nothing watches this" breaks the instant you list what is watching it. "Add a host-metrics exporter to one of the nodes" breaks when you find that node already runs a container metrics collector and ships its host metrics through a separate platform, so your exporter would double-count and page on itself.

The failure mode to design against is the opposite: an agent that takes the task at face value, builds exactly what you said, and hands you a clean diff for the wrong thing. A clean diff describes the code, not the world it runs in. Those four silently-failing units were the proof. Every one of them had been shipped after the monitor's coverage list was written, and the list was a static allowlist that rots the moment the estate grows. The highest-value item on the whole project turned out to be a watcher-of-the-wiring, an audit that asks whether newly-shipped units are actually covered. I never would have thought to ask for it. The recon found it by refusing to believe the task.

What makes a finish line a model can actually check?

The eleven files this run produced were never seen by the thing that graded it. A small, fast model scored the finish from the transcript alone, never running a command or opening a file. That one constraint rewrites how you write a done condition. "The eleven files exist and the tests pass" is unverifiable to that grader, so the run either loops forever or passes on the agent's word. "One final turn shows fresh command output and a self-audit table" is checkable in a single read.

An autonomous agent needs a terminal condition, the signal that tells the loop it is finished. The naive version describes the world: files present, tests green, service deployed. The problem is that whatever checks the condition is usually another model, and it has no eyes on your disk. It sees the transcript. A done condition phrased as a fact about the filesystem is therefore either uncheckable, so the run never converges, or taken on faith, so the agent can declare victory while nothing works.

The fix is to make the finish line transcript-demonstrable. Require the run to surface, in one final turn, the fresh output of real commands: the file listing, the diff, the passing test log, the grep that proves the wiring is live. Then require a self-audit table that maps each acceptance criterion to the evidence for it. Now the grader has something to grade. This is not ceremony. It is the difference between a run that ends because a model felt finished and a run that ends because it showed its work. Write the condition from the grader's real capabilities, not from a template you copied without asking what the grader can see.

Why gate the plan instead of the finished code?

Before any implementation, the recon collapsed nineteen candidate work-items into a design with exactly four genuinely-open decisions, and I resolved all four in a single structured prompt. Zero of the nineteen items collided with a project already in flight. The approval gate sat on the design, in the middle of the arc, not on the finished pull request at the end, because the middle is the only place an approval is cheap enough to be honest.

There is a strong pull toward letting an agent run start to finish and reviewing the pull request at the end. For infrastructure, that is the most expensive possible place to learn the plan was wrong. By the time a diff exists, the agent has committed to an architecture, and your choices are approve-with-misgivings or throw away real work. Reviewing a plan costs minutes. Reviewing and then unwinding a wrong build costs a day.

So the arc has one hard human gate, between research and implementation, and it is the only step that requires me. The agent produces a map: what it will build, what it deliberately will not, every prior decision it reconciled against, and a short list of the forks it genuinely cannot resolve alone. I answer the forks. Everything already settled is asserted plainly so it does not get re-litigated, and only the real decisions reach me. This is also where scope discipline lives. Nineteen items were dispositioned explicitly, each exactly once, so nothing was silently dropped and nothing raced work already running elsewhere. The plan is the artifact you can still change cheaply. Gate there, not at the end.

How do you make autonomy safe by construction rather than by prompt?

The remediation layer runs in three tiers, and none of them can take an action, because the capability to act was never written into any of them. The triage model sees only captured text and is never handed a tool it could call. The advice layer emits a proposal into a chat thread and holds no code path that executes. The incident-filing step can open a draft but has no path that closes. Each limit is structural, so a jailbroken or simply confused model hits a wall, not a moment of discretion.

"Propose-only" written in a system prompt is a request, and a request is not a safety property. The version that holds is the one where the capability is absent. The triage step is the clearest case. Instead of an agent with shell access reasoning about my servers, it is a deterministic collector that captures read-only output and a model that receives that text and returns structured JSON. The model cannot run anything, because nothing runnable was ever put in its hands. Its blast radius is the size of a JSON object.

The same shape repeats up the stack. The layer that suggests a fix writes the suggestion to a thread for a human to read; no branch anywhere in it executes the fix, so no prompt can talk it into one. The step that records an incident can draft a writeup, and the rule that an incident closes only after a human has confirmed the root cause and verified the fix is enforced by that step having no close path at all. This is graduated autonomy done honestly: each tier holds exactly the capability it has earned, and the tiers above it are not one clever sentence away. When you catch yourself trusting a model because you told it to behave, you are holding a prompt, not a gate.

Why isn't "armed and green" the same as working?

Every control that shipped was accepted by firing it, never by watching it sit healthy. We injected a synthetic fault and watched it travel the whole path to a chat thread. We ran one real triage cycle end to end and read the verdict it produced. We handed the refusal paths a case they should reject and confirmed they rejected it. A monitor that is armed and green has told you nothing at all until you have made it catch something.

The most seductive lie in operations is the green dashboard. An alerting rule with no alert, a remediation daemon with a healthy badge, a heartbeat that has never missed: all of them look identical whether they work or are quietly dead. "Armed and green" is a claim about a control's own status, not about whether it does its job. I have been burned by a reaper that reported healthy for weeks while reaping nothing, because nothing had ever forced it to act and reveal that it could not.

So the acceptance bar for the controls here was a fired proof. For the coverage audit, ship a unit it should flag and confirm the flag. For the triage path, run a real cycle and read the result. For the propose-only guarantee, feed it a case that must be refused and confirm the refusal returns an error instead of an action. The heartbeat is the honest exception worth naming: rather than trust a new bespoke alert, wire it onto the estate's already-proven dead-man monitor, the same one that pages when any other producer goes silent, so it inherits a firing that was tested long ago. A control you have watched catch a real event is a fact. A control you have only watched stay green is a hope wearing the costume of a fact. The extra hour spent writing synthetic faults is the cheapest insurance you will buy on infrastructure, because the alternative is discovering the control was decorative during the incident it existed to catch.

What do the five checkpoints add up to?

Across the project, five practices did the load-bearing work, and each one guards against a different way agent-built infrastructure fails silently. Not one of them depends on the model being smart. Every one of them is structure the human keeps: an argument before the build, a finish line a grader can read, a gate on the plan, capability bounded by construction, and acceptance by firing the control.

Checkpoint	The silent failure it prevents
Recon that refutes the task	building the wrong thing, cleanly
Transcript-demonstrable finish line	a run that ends on the model's own word
Approval gate on the plan, not the PR	learning the architecture was wrong after the work is done
Autonomy bounded by construction	a prompt talked into taking an action
Acceptance by firing the control	a decorative monitor that is green and dead

The through-line is that trust in agent-built infrastructure is not a feeling you develop about a capable model. It is a property of the harness around it. The model in this arc was the same model that wrote the confident, wrong first draft of the plan. What made its output trustworthy was that every step had a place where a wrong assumption got caught by something other than the model's own judgment. Recon caught the wrong premise. The grader caught the unfinished run. I caught the wrong plan. Construction caught the unsafe action. A fired test caught the decorative control.

Point a fast writer at your infrastructure with none of that, and you get exactly what the speed implies: the wrong thing, built well, running green, watching nothing. Keep the five checkpoints, and the agent becomes what it should be, a very fast pair of hands on a short and very well-lit leash. That is not a limit on what agents can do for operations. It is the thing that makes handing them the keys a sane decision instead of a hopeful one.

Originally published at danmercede.com.

Rekindle: your abandoned side project misses you 🔥

Dan Mercede — Fri, 10 Jul 2026 17:10:09 +0000

This is a submission for Weekend Challenge: Passion Edition

What I Built

Every developer has one: the side project you started with real love, pushed afew passionate commits to, and then... life. The tab closed. The flame wentquiet. Mine is spark, an intentionsetting app I started for my partner and me as the new year turned. One commit.190 days of silence.

Rekindle is a coach for exactly that repo. Paste a dormant GitHub projectand it gives you three things:

A diagnosis: why the flame died, read honestly from your commit cadenceand the gap between README ambition and what got built.
A rekindle plan: three steps, the first one deliberately small enough todo in under 15 minutes tonight.
A corner speech: a roughly 90-word hype speech about YOUR project, byname, delivered out loud like a cornerman between rounds.

The theme is passion. This is the tool for the week after the passion: the onethat gets you to open the project again.

Demo

Live app: https://rekindle-sooty.vercel.app

Try it on any public GitHub repo you've gone quiet on. If you just want to seea result instantly, this linkrenders a frozen real result (a live run against my abandoned spark repo,with the real ElevenLabs audio). Everything else is computed live per request.

Code

OrionArchitekton / rekindle

Your abandoned side project misses you. Gemini diagnoses why the flame died; ElevenLabs delivers the comeback speech. DEV Weekend Challenge: Passion Edition.

🔥 Rekindle

Your abandoned side project misses you.

Paste a dormant GitHub repo. Rekindle reads its README, recent commits, and top-level files, then delivers three things:

A diagnosis: why the flame died, read honestly from the commit cadence and the gap between README ambition and reality.
A rekindle plan: three steps, the first one small enough to do in under 15 minutes tonight.
A corner speech: a roughly 90-word hype speech about YOUR project, by name, spoken out loud like a cornerman between rounds.

Built for the DEV Weekend Challenge: Passion Edition because passion is also the love that fuels late-night side projects, and every developer has one whose flame went quiet.

How it works

GitHub URL --> GitHub API (README, commits, tree)
                 |
                 v
          Gemini 2.5 Flash (temperature 0, JSON output)
                 |  diagnosis + embers + plan + hype speech
                 v
          ElevenLabs TTS --> your

…

View on GitHub

How I Built It

Next.js 16 (App Router) with two server routes and a deliberately smallsurface:

GitHub URL --> GitHub API (README, last commits, file tree)
                 |
                 v
        Gemini 2.5 Flash (temperature 0, JSON output, thinking off)
                 |   diagnosis + embers + plan + hype speech
                 v
        ElevenLabs TTS --> the corner speech, out loud

Google AI does the reading. /api/rekindle snapshots the repo (metadata,README excerpt, last 10 commits, top-level files) and hands it to Gemini 2.5Flash with temperature: 0, responseMimeType: application/json, and thinkingdisabled, so output is fast, stable, and cheap. The interesting part wasthe failure I hit first: with thinking enabled, Gemini's reasoning tokens countagainst maxOutputTokens, so my 2048 cap silently truncated the JSON mid-fieldand my parser (correctly) rejected it. Thinking off + a bigger cap fixed it.The model output is schema-validated and length-clamped server-side; anythingmalformed renders as an error, never as half-parsed UI.

ElevenLabs does the believing. The hype speech only lands when you hear it.One catch: a public TTS endpoint on a server-side key is a free-for-all walletdrain, so /api/rekindle HMAC-signs the speech text it generates and/api/speak refuses anything unsigned (I verified the refusal with aforged-token test, and rate limits sit on both routes). Nobody gets to use mykey as a generic text-to-speech proxy.

Honest demo mode. /?demo=1 server-renders a frozen result so the demo isdeterministic. The frozen content is a real captured output from a live run,checked in with its matching real audio, and the page discloses it.

Unit tests cover the pure core (URL parsing, prompt build, defensive JSONparse). Built end-to-end in a Claude Code session (Claude Fable 5); the committrailers carry the co-author record, and I made the calls on idea, scope, andsubmission.

Prize Categories

Best Use of Google AI and Best Use of ElevenLabs: Gemini 2.5 Flashreads the repo and writes the diagnosis, plan, and speech as strict JSON;ElevenLabs (eleven_multilingual_v2) turns the speech into the voice in yourcorner. The two are chained: what Gemini writes about your specific repo isexactly what ElevenLabs says out loud.

Thanks for reading. Now go run npm install on the one you know I'm talkingabout. 🔥

Building Engram in 48 hours: what a memory engine taught me about forgetting

Dan Mercede — Fri, 10 Jul 2026 17:01:42 +0000

I entered the Qwen Cloud Global AI Hackathon's MemoryAgent track with a bias: I havespent months maintaining a typed memory system for my own AI agents (memory fileswith decay, an index under a hard size budget, contradiction notes). The track briefread like a checklist of the things that system taught me the hard way: efficientretrieval, timely forgetting, recall within limited context windows. So I productizedthe doctrine into an engine: Engram.

Three things earned their keep.

1. The budget is the feature. Most memory demos retrieve top-k and pray it fits.Engram scores every active memory (embedding similarity from text-embedding-v4,blended with retention, importance, and access frequency) and greedy-packs under ahard token budget, with backfill so a huge top memory does not starve the context.The packer has a property test over randomized pools: the selected set never exceedsthe budget. Writing that test first changed the design; the naive version failed it.

2. Forgetting needs a mechanism, not a cron job. Retention decays exponentiallyfrom last access with type-specific half-lives (episodes die in weeks, skills livefor months), scaled by importance. Recall refreshes retention, so what gets usedstays alive. Below the floor, a memory is marked decayed but never deleted; theaudit trail is the product too.

3. Contradictions are the hard 20 percent. New facts are checked against nearestneighbors and adjudicated by qwen3.7-plus at temperature 0 with strict JSON and afail-safe parser (a parse failure can never supersede anything). The subtle bug anadversarial review caught: my similarity gate was tuned too high, so "I moved toDenver" never even reached the adjudicator against "I live in San Diego" (they onlycosine at about 0.49 in 256 dims). Live probes, not intuition, set the threshold.

The meta-lesson: I ran an adversarial review pass that tried to refute 75 discreteclaims in my own README and submission against the code and live probes. It found aswapped argument pair feeding the adjudication prompt, the mistuned gate above, andhalf a dozen places where my wording was stronger than my evidence. An hour of beingrefuted is worth more than a day of polishing.

Engram is MIT-licensed, ships an MCP server so any agent can adopt it as a memorybackend, and runs on Alibaba Cloud with Qwen Cloud models end to end.

Code: https://github.com/OrionArchitekton/engram
Live: https://engram.orionbot.online

The Router Is a Trap: Running Other Models Under Claude Code, Off Their Own Subscriptions

Dan Mercede — Wed, 08 Jul 2026 22:17:23 +0000

I wanted a simple thing. Keep a Claude model orchestrating inside Claude Code, but when it fans out to subagents and background jobs, run that work on other models paid for by other subscriptions, so it stops drawing down my Anthropic allocation. The obvious build is a router: point Claude Code at a proxy, let the proxy translate each call to Gemini or GPT or a local model. That is the popular path, and it is a trap.

Two findings flipped my plan. The router path corrupts tool calls badly enough to be useless for agentic subagents. And the constraint that actually decides the architecture is legal, not technical: reusing a consumer subscription inside a third-party harness is prohibited and actively enforced at three of the four vendors I cared about. What survives both problems is the unglamorous option I already had running.

Who this is for: anyone trying to cut their Claude Code bill by offloading subagent or workflow fan-out onto other models, and anyone about to install a community "just point it at this proxy" bridge without reading what it does to your sandbox.

What are you actually trying to buy?

The goal is narrow and worth stating before the tooling: keep the frontier orchestrator (a Claude model) as Claude Code's primary driver, and route only the fan-out (subagents, workflow agent() calls, background tasks) to other models on their own billing. A Claude Code Max plan meters a weekly token allocation that resets on a fixed cadence, so every subagent you can move off that pool is allocation you keep for the orchestration that has to stay on Claude.

The key word is off-budget. This is the test every option below has to pass. Running a cheaper Claude model as a subagent does not pass it: Claude Code's native per-subagent selection is Anthropic-alias-only, and a Haiku or Sonnet subagent still bills the same Anthropic pool. Saving per-token cost is not the same as taking work off the allocation. Only routing to a genuinely external provider does that, which is why the interesting question is how to reach a non-Anthropic model at all.

Why does the router path corrupt your tool calls?

The most-starred tool in this space, claude-code-router, sits at roughly 35,600 GitHub stars as of mid-2026, and it works by exploiting one documented fact: Claude Code does not validate model IDs behind a custom ANTHROPIC_BASE_URL. Anthropic's own model-config docs say the ID check "runs only on the Anthropic API," so behind a gateway your provider defines the names and Claude Code passes any string through unchecked. That single gap is what every under-the-harness router is built on.

It works at the transport layer and fails at the semantic one. The dominant, recurring failure is tool-calling corruption. When a non-Claude model runs behind Claude Code's Anthropic-shaped API, the translation shim mistranslates tool calls: optional parameters get injected as empty strings (a Read arrives with pages: ""), which Claude Code rejects, "producing an infinite loop that burns reasoning tokens until the user manually interrupts." Server tools like web search get mistranslated into ordinary functions, so search comes back empty or misleading. Some local-model adapters emit plain text that merely mimics a tool call and then does nothing with it. On top of that, documented proxy layers drop MCP server tools, ignore prompt-cache control, and turn parallel tool use into sequential calls.

For a chat turn, that degradation is survivable. For an agentic subagent whose entire job is to call Read, Edit, Bash, and MCP tools reliably, it is disqualifying. The proxy gives you the model behind the endpoint and takes away the tool loop that made the endpoint worth reaching.

Why is the Terms of Service the real constraint, not the tooling?

Here is the finding that actually decided the architecture: as of mid-2026, reusing a consumer subscription inside a third-party harness is prohibited and enforced at three of the four vendors, so the tooling question is downstream of a legal one. The enforcement is not theoretical, and in one case it is not even loud.

Anthropic made the prohibition explicit in a February 2026 consumer-terms revision (tokens permitted only in Claude Code and Claude.ai), and on April 4, 2026 blocked third-party harnesses from using Claude Max subscription rate limits outright. Bring-your-own API keys still work; subscription-OAuth reuse does not.
Google is the cautionary tale. It deployed abuse detection and silently banned paying AI Ultra subscribers (a $249.99-per-month plan) who drove third-party CLIs on their consumer OAuth in mid-April 2026, with reports of the ban cascading to Gmail and Workspace. It then stopped serving consumer and Pro or Ultra tiers to its IDE code-assist extensions on June 18, 2026. If your estate leans on Google Workspace, an OAuth-reuse ban is not a coding-tool outage, it is a Gmail outage.
xAI's terms bar automated, non-human (bot or script) access to Grok and reserve suspension at sole discretion; the only sanctioned programmatic path is the separate pay-per-token API.
OpenAI is the outlier. It publicly sanctions reusing a ChatGPT subscription in third-party tools with "the same usage as Codex," and Codex-into-third-party has an official adapter. Subscription Codex is governed by the consumer ChatGPT terms and metered as plan usage, not raw API billing.

The net for the original dream of "route all my subscriptions through one proxy": it is durable for exactly one of them, GPT via Codex. Every other subscription seam rides reverse-engineered OAuth that the vendor is actively closing. One corroborating datum: a Claude-Max-as-API seam that made the rounds earlier in 2026 survived about two months before the vendor shut it. Treat any subscription-brokering feature as a liability with a short shelf life, not a capability. For Gemini or Grok specifically, use a paid API key, never the consumer login.

What actually works: shell out to headless CLIs

The mechanism that clears both the tool-fidelity trap and the Terms-of-Service wall is the one I already ran in production for months: keep Claude orchestrating and shell out to each other vendor's headless CLI as a subagent. Instead of translating Claude Code's API to another provider, you invoke codex exec, gemini, or grok -p as a subprocess, each authenticated to its own login, each doing its own native tool calling.

This inverts every problem the proxy created. There is no Anthropic-to-OpenAI translation shim, so there is no empty-string-parameter loop and no mistranslated web search; each CLI speaks its own protocol natively. The work runs on that CLI's own credentials, so a Codex call bills the ChatGPT plan and a local-model call bills nothing, both off the Anthropic allocation. And it matches a doctrine most mature harnesses already follow, which is to prefer a working CLI over a bespoke integration: a post-push review pipeline that drives Codex headless is the same pattern, just pointed at review instead of fan-out.

The tradeoff is honest. You give up the illusion of one unified /v1 endpoint and instead orchestrate N processes, joined by your own wait-and-collect logic. In exchange you get native tool loops, per-vendor billing, and no dependency on a subscription seam that gets patched out from under you.

How do you keep the delegation layer from bypassing your sandbox?

The security catch is real and specific: nearly every community CLI-bridge I looked at launches its child CLI with a permission-or-sandbox bypass flag baked in (--dangerously-skip-permissions, --dangerously-bypass-approvals-and-sandbox, --yolo, -y). That is convenient and it is also how you hand an untrusted subagent's output the keys to your workstation. When I shipped my own delegation layer in July 2026, the rails were the point, not an afterthought, and they are enforced in code with tests, not just documented.

Five rails carry the safety, and each is one small function:

Never emit a bypass flag. A denylist is asserted in a unit test so a bypass flag can never regress into the spawned argv. The read-only sandbox is the default; write access is an explicit, separate opt-in.
Scrub the child environment by whitelist, not denylist. The child gets PATH, HOME, and its own provider key, and nothing else. A future secret-shaped variable you have not thought of yet does not survive by default, because the default is to drop, not to keep.
Resolve the CLI by absolute path. Never exec a bare command name; resolve it to an absolute path first so a planted binary earlier on PATH cannot shadow the real one.
Close stdin. Recent Codex versions deadlock on a non-TTY pipe, so the child gets stdin pointed at /dev/null.

# read-only by default; workspace-write is an explicit, separate opt-in
sandbox = "workspace-write" if write else "read-only"
argv = [resolve_cli("codex"), "exec", "-s", sandbox, task]
assert not any(flag in argv for flag in BYPASS_FLAGS)  # tested, not just asserted here

The fifth rail I only found by running the thing, which is the reminder that live verification beats reasoning. Grok's CLI needs an explicit headless flag: grok -p "<prompt>" prints the answer and exits, but a positional prompt with no flag opens the interactive TUI and hangs a non-interactive caller. A positional prompt cost me a two-minute timeout before I read the help text. The regression is now pinned by a test that asserts the -p flag is present in the built argv.

# a positional prompt opens grok's interactive TUI and hangs a non-TTY caller;
# -p / --single prints and exits.
argv = [resolve_cli("grok"), "-p", task, "--output-format", "json"]

What should you actually build?

If you take one architecture from this, take this shape, which I settled on after two research runs and a working implementation in July 2026: Claude orchestrates, and it delegates to a small, gated set of headless CLIs chosen by Terms-of-Service durability first and cost second.

Orchestrator: a Claude model in Claude Code, unchanged.
Primary off-budget subagent: GPT via codex exec on the ChatGPT plan. This is the one durable subscription-reuse path, so it earns the default slot.
Cheap bulk fan-out: a metered API key (DeepSeek and Kimi run roughly an order of magnitude cheaper than frontier Claude on input tokens) or a free local model on your own GPU box via a local inference endpoint. No subscription to reuse means no ban risk.
Gemini or Grok, if at all: only on a paid API key, isolated from the primary Google account. Never the consumer login.
Mechanism: headless-CLI subprocess, not a proxy. If you ever want a proxy for API-key models, the routers are fine; just treat their subscription-brokering as the liability it is.

The three paths compared, so the choice is legible:

Feature	Router proxy	Headless CLI (recommended)	Native per-subagent
Keeps Claude orchestrating	Only if you patch aliases	Yes, cleanly	Yes
Routes subagents to other providers	Yes, but Claude-unaware	Yes, per call, model named	No, Anthropic aliases only
Off the Anthropic budget	Yes	Yes	No, still bills the same pool
Tool-calling reliability	Poor and brittle	Native per CLI	Native
Subscription legality	Rides the prohibited seam	Depends on vendor (GPT sanctioned)	Not applicable

The plan I walked in with, a single elegant router, was the wrong instinct dressed up as the clean one. The winning design is a handful of subprocesses behind a security gate, choosing each model by whether its vendor will still let you in next month. Boring, durable, and off-budget. That is the trade, and it is a good one.

The vendor terms and enforcement dates in this guide are time-sensitive and were accurate to mid-2026; enforcement is active and moving, so re-verify a vendor's current terms before you build on any subscription seam.

Originally published at danmercede.com.

Giving Your Coding Agent Web Access Without Wrecking Its Context Window

Dan Mercede — Mon, 06 Jul 2026 03:30:59 +0000

Most coding agents get web access the lazy way: a fetch tool that returns raw HTML, a search tool wired straight into the main context, and a metered API behind both. It works in a demo. Then the agent reads three documentation pages, and half its context window is <div class="nav"> and inline analytics.

I spent a day hardening the web-access layer for my own agents and measured every option on two axes almost nobody ranks first: cost and rate-limit resilience. The result is a stack that is free or self-hosted for the load-bearing paths, plus one structural pattern that matters more than any single tool: keep raw pages out of the main context by design.

This guide is the version I wish I had at the start. Every number is from a public source or a run you can repeat.

Who this is for: anyone giving an autonomous agent (Claude Code, a -p headless pipeline, Codex, an in-house harness) the ability to search, read, and drive the web, and paying for the tokens.

Why does giving an agent web access wreck its context window?

Because raw HTML is mostly not content. In one measured fetch, the same page was 9,541 tokens as raw HTML and 1,678 tokens as clean markdown, an 82% reduction. Typical HTML-to-markdown conversion lands nearer 20 to 30%, but the direction is constant: you are paying to put boilerplate into the model's attention, then paying again on every following turn.

The waste compounds three ways. Context flood: a large docs page is roughly 25,000 tokens of input, and it stays resident for the rest of the conversation. Cost: search results and fetched content are billed as input tokens on the turn they arrive and every turn after, so a page you read once is re-billed until it ages out. Rate limits: point a fan-out of parallel agents at a metered search API and you hit 429 almost immediately. The lazy path fails on all three at once.

Context rot is the quiet one. Even on a large context window, recall degrades as the window fills. A page of HTML you never needed is not free just because it fit.

What are the three axes of agent web access?

Web access is not one capability, it is three, and they want different tools. Search (find the URLs), fetch (read a page without drowning in it), and browser automation (click, fill, scroll a live app). In a 2026-07 review of the field, the cheapest resilient answer differed per axis: self-hosted for search and fetch, local-and-free for the browser.

Conflating them is how people end up overpaying. You do not need a headless Chromium to read a blog post; an extract-first markdown fetch handles the large majority of retrieval. You do not need a metered search API to read a page whose URL you already have. Name the axis, then pick the cheapest tool that covers it. Reserve the expensive, stateful browser for genuine interaction: clicks, forms, infinite scroll, auth walls.

The rest of this guide takes the axes in order, cheapest and most rate-limit-resilient first.

What is the cheapest, most rate-limit-resilient search stack?

Self-hosting wins outright. A self-hosted SearXNG metasearch instance runs on a 5 to 10 dollar per month VPS, aggregates roughly 95 to 200 engines (Google, Bing, Brave, DuckDuckGo and more) behind a JSON API, and charges nothing per query. Put a Valkey or Redis cache in front of it, because individual upstream engines rate-limit a single client IP hard, and the cache is what absorbs that.

If you want a hosted API instead, rank by the free tier and the rate cap, not the marketing. Exa gives 20,000 free requests a month. Brave gives about 1,000 free queries a month but caps the free tier at 1 query per second, which breaks the instant you do parallel fan-out search, a common agent pattern. Perplexity Sonar and Kagi are metered with no meaningful free tier. Anthropic's native web_search is the priciest per call at 10 dollars per 1,000 searches plus token costs, though its companion web_fetch adds no per-fetch fee and ships a max_content_tokens cap worth using.

One honest caveat carried from the research: a "Serper plus a reader" stack was reported at about 11 dollars a month for 10,000 searches and 10,000 extractions, against roughly 96 dollars for a single-vendor API on the same workload. That specific figure rests on a single blog and should be read as directional. The ranking (self-hosted free, then Exa/Brave free tiers, then metered) is the corroborated part.

What is the cheapest way to read a page without flooding context?

Extract-first, markdown, in that order. The single biggest lever is converting a page to clean markdown before it reaches the model, which cut tokens 82% in the measured case above. Jina Reader is the cheap default: prefix any URL with the reader endpoint and you get markdown back, free at 20 requests per minute with no key, or 10 million tokens on a free key, and it is open-source so you can self-host it. Trafilatura is a self-hosted Python extractor with no API cost and strong extraction quality, though it will not render JavaScript.

For scraping proper, the same free-first ranking holds. Crawl4AI is Apache-2.0 with no per-page fees; your only cost is the box it runs on. Managed Firecrawl trades that for convenience at roughly 83 dollars a month on its standard tier. The crossover is volume: at low and mid volume the managed credits are cheaper than running infrastructure; past high volume, the self-hosted crawler wins. Pick the tool, but do the markdown conversion no matter which you pick. Handing a model raw HTML is the mistake; the extractor is incidental.

Why does a thin CLI beat an MCP server for agent web tools?

Because an MCP server costs context on every host, and a CLI costs almost none. As of 2026-07, Claude Code defers MCP tool schemas until a tool is invoked, so an MCP is cheapish there. But Codex has no such deferral: every MCP schema loads in full, every session, and a harness that already carries a dozen servers pays that tax constantly. A thin CLI you shell out to via Bash costs zero resident context until the moment you call it.

So for anything I want available on more than one agent host, I ship a CLI, not an MCP. I built three this way: a search CLI over a grounded-answer API, a live-social search CLI, and a page-reader over Jina. Each self-wraps its own secret retrieval (read the key from the environment, or transparently re-run under a secrets manager if it is not set) so the caller passes no credentials, and each takes a --json flag so an agent can parse it. One implementation, every host, near-zero context. The rule generalizes: prefer a working CLI over its MCP unless the tool genuinely needs a stateful session.

Browser automation is the one real exception. Playwright, driving a real browser, is stateful and does not reduce cleanly to a stateless CLI, so it stays an MCP or an in-process library. There, the free-and-local option still wins: Playwright is Apache-2.0, runs on your own machine at zero LLM cost, and hits about 98% reliability on deterministic scripts. Running Playwright's MCP in its CLI mode reportedly cuts token use about 4x versus the protocol mode. Metered cloud browsers exist (Browserbase, Stagehand's hosted tier) but the free tiers are thin (Browserbase caps the free tier at 1 browser-hour, 3 concurrent sessions, 15-minute sessions), so reserve them for scale you have actually hit.

How do you build the fetch tool, and why run it in a subagent?

Because the tool only wins if the raw page never reaches the main agent at all. The pattern is small: a fetch CLI that returns bounded markdown, invoked inside a subagent whose job is to read the page and return only the distilled slice the main agent asked for. The main context sees a paragraph, not 25,000 tokens. Even the markdown never lands in the parent conversation.

The fetch CLI itself is about 120 lines. The load-bearing parts:

import math, os, urllib.request  # the full CLI also imports json (for --json) and sys

READER = "https://r.jina.ai/"
# The default python-urllib User-Agent gets a 403 from the reader's bot filter.
# Send a real browser UA. This one line is the difference between 200 and 403.
UA = os.environ.get("WEB_READ_UA",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/126 Safari/537.36")

def fetch(url, key=None):
    req = urllib.request.Request(READER + url, method="GET")
    req.add_header("User-Agent", UA)
    req.add_header("X-Return-Format", "markdown")
    if key:                                  # optional, raises free-tier limits
        req.add_header("Authorization", f"Bearer {key}")
    with urllib.request.urlopen(req, timeout=60) as r:
        return r.read().decode("utf-8", "replace")

def budget(md, max_tokens):
    # A stdlib token estimate: ~4 chars per token. Label it estimated.
    est = math.ceil(len(md) / 4)
    if not max_tokens or est <= max_tokens:
        return md, est, False
    return md[: max_tokens * 4].rstrip() + "\n\n... [truncated]", max_tokens, True

Three details earn their place. It is keyless by default (the reader's free tier needs no key), so the tool works with zero setup and only reaches for a key to raise limits. It carries a token budget (--max-tokens), so a runaway page cannot blow the context even inside the subagent. And it emits --json so the calling agent gets structured output, not a wall of text. The browser User-Agent is the one non-obvious line: a live smoke test, not the unit tests, is what caught the reader returning 403 to the default python-urllib agent. Unit tests pass with a mocked network; only exercising the real endpoint surfaces that class of bug.

What should you actually adopt?

Start free and self-hosted, add metered tools only where you have measured a need. As of 2026-07 the stack I run is: self-hosted SearXNG behind a cache for search, a keyless Jina-backed reader CLI for fetch (run in a subagent so raw pages never hit the main context), a self-hosted crawler for bulk scraping, and local Playwright for the genuine browser work. Native provider search sits behind all of it as a metered last resort.

The through-line is not a specific tool, it is two rules. Convert to markdown before the model sees a page, which reclaims most of the token bill on day one. And offload the fetch to a subagent, so the raw content, even cleaned, is summarized down before it touches the conversation the main agent has to keep coherent. Everything else is a cost-and-rate-limit optimization on top of those two moves.

The takeaway

Giving an agent web access is easy; giving it web access that does not quietly bankrupt its context window takes two deliberate choices. Extract to markdown, and read in a subagent. Do those, rank every tool by free-tier and rate cap before you reach for a paid API, and prefer a thin CLI over an MCP for anything that is not a stateful browser. The lazy path costs you a large slice of your web-read tokens, up to 80% on a boilerplate-heavy page, and a 429 under load. The fix is a day of plumbing and a 120-line script, and it pays for itself the first time your agent reads the docs.

Originally published at danmercede.com.

The Reviewer Blessed the Bug: Why Agent-Written Code Needs Layered Review

Dan Mercede — Sat, 04 Jul 2026 15:56:53 +0000

A security reviewer read my diff and signed off on a specific claim: the credential-scrubbing was correct, "encoded access/secret forms are distinct, no overlap mis-redaction." It was thorough. It traced every log path. It was also wrong.

One review layer later, a different reviewer took the same function, fed it two overlapping credentials, and printed the result: a partial secret sitting in the log line. The redaction had eaten part of one secret and left the tail of another exposed. Same code, same afternoon, opposite verdict.

That gap between "a careful reviewer certified this" and "an independent adversary broke it in one run" is the entire case for layered review. When an agent writes your code, and increasingly it does, no single review pass is a safety net. The passes have to be independent, because independence is the only thing that makes their blind spots not line up.

This guide walks the real arc: one small log-scrubber, four review layers, five defects, and the exact modality that let each layer catch what the others could not see. Every bug here is a shape you can copy into your own checks.

Who this is for: anyone shipping code an AI agent wrote, or building the review harness that gates it, especially around self-hosted infrastructure where a quiet failure hides for weeks.

What was actually being hardened?

In a 2026-07 pass on a self-hosted LLM-observability stack (Langfuse on top of ClickHouse, with backups pushed to a MinIO object store and an encrypted off-box replica via restic), I hardened one function: a log scrubber that strips credentials out of backup-failure messages before they reach a cron log. Roughly 20 lines. It is exactly the kind of small, security-adjacent code an agent produces in seconds and a human skims in seconds.

The scrubber matters because backup scripts fail loudly, and a failure message often contains the command that failed, which contains the credentials. If a nightly backup dies and dumps an S3 secret key into a log that ships to your aggregator, you have turned a reliability event into a credential-exposure event. The scrubber's whole job is to make that impossible.

So it is a good test case: tiny, high-stakes, and the sort of thing everyone assumes is too small to need real review. It went through four layers anyway. Here is what each one saw.

Why does live-environment recon catch what no diff can?

Before any diff review, I ran the backup once against the real host, and that single run exposed a defect no code reader could have found: the script selected its MinIO container with a fuzzy name match, and the host happened to run two MinIO containers. The match grabbed the wrong one, hit Access Denied on a bucket it could not see, and reported a false CRITICAL every night while the actual backup was fine.

No diff review catches this. The code was locally correct. docker ps filtered by a name substring returns the first match, and on a one-MinIO dev box that first match is always the intended one. The bug only exists in the shape of the live environment, where a second, unrelated MinIO happened to sort ahead of the one the backup wanted. The lesson is old and keeps being true: a passing local test and a clean diff describe the code, not the world it runs in. The fix was to select the container by exact name, defaulted in-script, so a fuzzy match can never wander onto the wrong instance.

Recon is a review layer in its own right. It senses runtime state, a modality a static reviewer cannot reach. If your agent-review harness is all diff-readers, it stays blind to an entire class of defect that only appears on contact with production.

What does a pre-PR reviewer fleet catch that a single reviewer misses?

Before the pull request opened, I ran three independent reviewers over the diff (adversarial, correctness, and security), and the security pass caught a redaction bug hiding in plain sight: the scrubber replaced each secret with a shell pattern substitution, s="${s//${SECRET}/<redacted>}", and the unquoted needle is treated as a glob. A secret containing a bracket expression like [ab] is read as a character class, fails to match itself literally, and passes through the log unredacted. A * or ? in the secret fails the other way, over-matching and redacting adjacent log text. Both are wrong; only one leaks, and you cannot predict which without knowing the secret.

The fix is one edit, quoting the needle so the match is always literal:

# unquoted: a bracket-expression secret leaks, and * or ? over-redacts
s="${s//${SECRET}/<redacted>}"
# quoted: the secret is matched literally, both failure modes closed
s="${s//"${SECRET}"/<redacted>}"

The value of the fleet is not three times the reading. It is three different readings. Point one reviewer at a diff and you get one model's priors about what is worth checking. Point three with distinct adversarial framings and their blind spots do not fully overlap, so the union of what they notice is strictly larger. This is the cheapest layer to add and the one people skip most, because a single "looks good" from a capable model feels like enough. It is not enough. It is one sample.

Why did the post-push layer catch what the security reviewer had certified?

With an access key of ABC and a password of ABCDEF, a post-push reviewer ran the already-certified scrubber on the line password=ABCDEF and got back password=<redacted>DEF: a partial secret in the log. The pre-PR security pass had checked this exact function for overlapping-secret bugs and signed off, "encoded access/secret forms are distinct, no overlap mis-redaction." A second model, running the code instead of reasoning about it, broke that certification in one run.

Here is the whole bug. The scrubber redacted each secret in sequence. If a shorter secret is a substring of a longer one, the shorter replacement fires first, mutates the line, and the longer secret no longer matches itself, so its tail survives:

# ACCESS='ABC'  RESTIC='ABCDEF'
s="${s//"${ACCESS}"/<redacted>}"   # 'password=ABCDEF' -> 'password=<redacted>DEF'
s="${s//"${RESTIC}"/<redacted>}"   # no longer matches; 'DEF' leaks

The fix is to stop treating the needles as an ordered list and redact them longest-first, so no shorter needle can fragment a longer secret before that secret is matched whole:

# sort the needles by length, descending, then redact each literally
if [ ${#needles[@]} -gt 0 ]; then
  readarray -t needles < <(printf '%s\n' "${needles[@]}" | awk '{print length"\t"$0}' | sort -rn | cut -f2-)
  for n in "${needles[@]}"; do s="${s//"${n}"/<redacted>}"; done
fi

Longest-first closes the substring case, where one secret sits wholly inside another. It does not close the rarer case of two distinct secrets that overlap in the log line, a suffix of one equal to a prefix of another; random credentials almost never do this, but if yours can, redact in a single combined pass instead of a loop. State the bound; do not sell the fix as total.

This is the load-bearing section, so sit with what happened. A competent reviewer did not skip the overlap question. It considered the overlap question and answered it wrong, because it reasoned about the encoded forms being distinct and missed that the raw forms could overlap. A second, independent adversary, running the function instead of reasoning about it, found the counterexample immediately. Certification by one reviewer is an argument. Reproduction by an independent one is a fact. When you can afford only one, prefer the layer that runs the code.

Can a merge gate itself be a review layer?

The last defect was not in the code at all, and the CI gate surfaced it: my new test used credential-shaped fixtures (values with an AKIA prefix, the AWS access-key-ID shape), and a secret scanner flagged them and transitively blocked the merge, even though that scanner was not one of the branch's required status checks. A second scanner, run on the identical fixtures, passed. The gate disagreed with itself.

The trap is one layer of indirection. The scanner was not in the branch-protection required list, so it looked advisory. But a separate fail-closed meta-check that was required carried its own internal list of checks it waits on, and the scanner was in it. So a "non-required" check gated the merge through a required one. Reading only the branch-protection settings would tell you the merge was clear. It was not.

Two portable lessons fall out. First, never hardcode credential-shaped fixtures; use obviously-fake, low-entropy placeholders that still exercise the code path, because a scanner cannot tell your test data from a live leak and should not have to. Second, when a merge is blocked and the required checks look green, read what your meta-gates actually wait on, not just the protection rules. The scanner also scans full branch history, so neutralizing the fixtures at the tip does not clear a hit an earlier commit introduced; a squash-merge that collapses to the net diff does.

Why don't the layers just catch the same things?

Across the arc, four layers caught five distinct defects with no overlap in the catches (the merge gate alone surfaced two, the credential-shaped fixtures and the transitive block), because each layer senses a different modality. Live recon senses runtime state. The pre-PR fleet senses the diff with fresh adversarial priors. The post-push pass senses by an independent model that runs the code. The CI gate senses policy and scanner state. Independent modalities have independent blind spots, which is the only reason stacking them pays off.

Review layer	Sensing modality	The defect it caught
Live recon	runtime state on the real host	wrong container selected, false CRITICAL nightly
Pre-PR reviewer fleet	fresh adversarial read of the diff	glob-unquoted redaction needle
Post-push pass	an independent model that runs the code	sequential-substitution partial-secret leak
CI merge gate	policy and scanner state	credential-shaped fixtures, transitive merge block

Stacking identical reviewers does not work. Three passes from the same model with the same framing tend to miss the same things, because they share priors. The gain comes from orthogonality: a runtime probe sees what a reader cannot, a code-runner sees what a reasoner cannot, a scanner sees what a designer forgot. The arc is a clean natural experiment for this, because one layer certified the precise property another layer broke. If the two had shared a modality, they would have shared the blind spot, and the leak would have shipped.

The uncomfortable corollary for AI-written code: the model that wrote the code, the model that reviews the diff, and the model that reasons about correctness are correlated failure sources when they are the same model reasoning the same way. Independence is not a nice-to-have on top of review. Independence is the review.

How do you build this without drowning in process?

You do not need a 20-layer gauntlet on every one-line change; you need a few genuinely independent layers on anything security-adjacent, and a smaller set elsewhere. In practice, for agent-written infrastructure code, four layers cover the modalities that matter, and each is cheap once wired.

Concretely:

Run it against the real environment before you trust the diff. One live execution surfaces the wrong-container, wrong-endpoint, and "works-on-the-dev-box" class that no reader catches. Recon is a layer.
Fan out a small pre-PR reviewer set with distinct framings, not one reviewer three times. Adversarial, correctness, and a security pass on the named bug classes. The union beats the average.
Add one post-push pass by a different model that runs the code, not just reads it. This is the layer that catches what your best reviewer confidently blessed. Weight reproduction over certification.
Treat the CI gate as a reviewer and read its real dependency graph, including what your fail-closed meta-checks wait on. And keep credential-shaped data out of tests entirely.

The through-line is not "more review." It is independent review. Five defects in one 20-line function, each invisible to every layer but one, is the normal shape of software when a fast writer meets a single slow reader. That is not a verdict on the agent; it is arithmetic. Give the writer four orthogonal readers instead, and the blind spots stop lining up.

Originally published at danmercede.com.

Building a Governed, Double-Send-Safe Delivery Pipeline for Agent Outputs

Dan Mercede — Fri, 03 Jul 2026 22:17:35 +0000

I built a multi-agent system to run a small business. Agents drafted work, and some of that work left the building: emails to real people, exported documents, delivered artifacts. I later retired the business on market grounds, but the delivery pipeline is the piece I would rebuild without changing a line. It answers a question most agent demos skip: how do you let an autonomous system send things on a person's behalf without ever double-sending, misfiring, or shipping something nobody approved?

This guide is the pattern, generalized and stripped of the business it ran. It is for anyone whose agent sends consequential output: customer email, outbound messages, exported records, anything where "sent twice" or "sent without sign-off" is a real cost rather than a cosmetic bug.

The one-line version: treat every outbound action as a durable, leased, fail-closed transaction with a human approval gate and an evidence receipt, and make the crash-recovery path refuse to re-send by default. The rest of this guide is that sentence, unpacked.

Why does an agent that sends email send it twice?

The double-send is not a rare edge case; it is the default behavior of a naive queue the first time a worker dies mid-send. A worker claims a send task, hands the message to the mail server, and then is killed (out of memory, a deploy, a reboot) before it writes "done." The task still looks unfinished, so the next worker picks it up and sends again. The customer gets two emails and you get a support ticket.

Every safe-delivery decision below exists to close that window. The failure is structural, not bad luck: the moment of maximum danger is the gap between "the side effect happened in the outside world" and "the system recorded that it happened." A crash in that gap is when a retry becomes a duplicate. You cannot make the gap zero, so you design for a crash inside it.

What does the governed delivery pipeline look like?

The pipeline composes into one spine, and the rule is that no consequential send skips the approval, single-send, and attest gates. The score and review stages decide which items a human reads first rather than gating every send. Six stages, each durable before the next begins:

Produce. The agent generates the artifact in full (for email, a complete RFC 5322 message) and stages it. Nothing is sent inline at generation time.
Persist. Write the intent (the staged artifact plus a queue row and a receipt) to durable storage before anything leaves the process. If you crash now, the intent survives and is inspectable.
Score. Attach a confidence score to the output. This is the input to review routing, not a gate by itself.
Review. Route low-confidence items to a human review queue. High-confidence items still do not bypass the next stage.
Approve. A fail-closed approval gate. The release is blocked by default and proceeds only on an explicit, recorded authorization.
Send and attest. Send exactly once under a lease, then reconcile and write an evidence receipt describing what happened.

The important property is that steps 2 through 6 are separate durable transitions, not one function call. Each stage can crash and resume without losing or duplicating work, because the state that matters lives in the store, not in a worker's memory.

How do you make the send idempotent when a worker dies?

This is the stage most teams get wrong, and the fix is a rule, not a library. Use row-level leasing so exactly one worker owns a task at a time, and make external-send actions non-requeueable on lease expiry. In Postgres the leasing primitive is a claim under SELECT ... FOR UPDATE SKIP LOCKED: concurrent workers select different rows, never the same one, so two workers cannot both send.

The lease is what makes crash recovery safe, and the reaper is where the real decision lives. A reaper reclaims tasks whose lease expired because the worker died. For ordinary internal work, reclaiming and re-running is correct. For an external send it is a loaded gun: if the worker died after the mail server accepted the message but before it recorded success, re-running sends a duplicate. So the rule is:

An external-send action is non-requeueable on lease expiry. When its lease dies, the reaper strands the task for human reconciliation instead of re-running it.

That is the whole trick. A visible stuck task beats an invisible double-send. You trade an automatic recovery you cannot prove is safe for a manual one a human can resolve in seconds by checking whether the message actually went out. Internal, side-effect-free tasks stay auto-requeueable; only the actions that touch the outside world are stranded. Encode the action type in the task and let the reaper branch on it; do not rely on a human remembering the distinction.

How do you keep the pipeline fail-closed?

Fail-closed means the safe state is the default and every unsafe action requires an explicit, positive signal to proceed. Three defaults carry most of the weight:

Sending is off until switched on. A single delivery-enabled flag gates all outbound sends, defaulting to off. A misconfiguration, a fresh environment, or a half-deployed release sends nothing rather than sending wrong.
The sender identity is checked, not assumed. At the point of send, the from-address is checked against a verified sending domain, and a transport guard refuses to send if encrypted transport is expected but the connection would fall back to an unencrypted or wrong port. These checks live in send code, not in a config comment.
Nothing consequential ships without a recorded approval, and everything that ships leaves a receipt. The approval gate is the choke point; the receipt is the audit trail. A stranded, unreconciled send is deliberately left without a success receipt, because its outcome is genuinely unknown until a human checks.

A short operational note, because it cost me. A swallowed import error once left a preflight check silently broken for about 65 days; a reboot finally surfaced it, and the whole time the failure had been read as a benign "dependency unavailable" state. A suppressed error that reads as a normal state is its own failure mode. Fail-closed only helps if the closed state is loud.

When do you actually need this, and when is it overkill?

Match the machinery to the blast radius of a wrong send. For low-stakes internal output (a draft in a channel, a log line, a suggestion a human will obviously see and fix), this full pipeline is overkill and you should not build it. The cost of the ceremony exceeds the cost of the mistake.

You need it the moment a wrong or duplicate send has real cost: customer-facing email, anything touching money, anything with a compliance or legal record, anything a recipient acts on. For those, the pipeline is the floor, not the gold-plating. The tell is simple: if "sent twice" or "sent without sign-off" would generate a support ticket, a refund, or a liability, you are in scope.

The pattern outlived the business I built it for, which is the honest reason it is here rather than in a product. The decision to retire that system, and the principle that a clean architecture is a reason to document the pattern rather than keep funding the business, is its own argument, made in the companion essay "Architecture Fit Is Not Business Justification." Build the pipeline when the sends are consequential. Skip it when they are not. And whatever you build, make the reaper refuse to re-send by default.

Originally published at danmercede.com.

The Alibi of a Green Gate

Dan Mercede — Tue, 30 Jun 2026 00:28:56 +0000

A control is what it does to the live target, not what it is declared to be. We build the gate, the reaper, the required check, the backup. We arm it, it runs, it exits zero, the dashboard is green. Then the leak it was meant to kill is still leaking, the bad merge it was meant to refuse already landed, the stale image it was meant to replace is still serving months-old code. Present, configured, armed, inert. None of those four facts is the one that matters.

The failure has one shape, and it is structural. The control's selection model, its liveness model, or its notion of "done" does not match the world it governs, so it sweeps an empty set, or whitelists by naming the bad states it knows and silently passing the ones it does not, or inspects the merged PR instead of the running artifact. Nothing throws. A no-op and a success produce the identical green. The green proves the control ran; it says nothing about whether it bit.

That is the whole trap: "armed and error-free" reads exactly like "remediating," and the reading is free while the proof costs a probe. So pay for the probe. The question is never "did it run." It is "did anything in the live world change because it fired": did the leak shrink, did the merge get refused, did the served bytes turn over. A control that cannot point at an effect on its target has not earned the word "working." Trust the effect, never the exit code. A green gate over an inert control is theater. Prove it bit.

Originally published at danmercede.com.

Coherent Isn't Correct

Dan Mercede — Mon, 29 Jun 2026 00:30:00 +0000

Coherent Isn't Correct

Experiment Log · 2026-06-28 · 05:30 PM PT

Hypothesis: Hand an autonomous AI coding agent a whole local-first finance tool to build end-to-end, no human writing code, and the output will be safe to use as-is.

Constraint: First pass fully autonomous; then an independent audit and a harden loop where every finding becomes a failing test before it's fixed; every public claim must be backed by code.

Result: Failed. The agent (Grok) shipped something genuinely coherent — an installable, Dockerized, CI-green vertical with a CLI, a web UI, a sqlite ledger, invoices, and reports. But it was broken in exactly the parts that matter for money: amounts stored as floats (the Markdown and JSON reports disagreed), a fabricated 42-dollar charge recorded on a receipt it couldn't read, a parser that truncated 1,234.56 down to 1.23, a 'CSV export' that emitted JSON with no OFX at all, an 'optional local vision' feature with zero code behind it, and a file upload with a path-traversal hole that ran as root. The harden loop — a self-adversarial review plus three independent automated review passes, each finding turned into a failing test — caught a critical data-loss bug in the upgrade path and fixed the rest. v0.2 now uses exact integer money, flags unreadable receipts for review instead of inventing amounts, ships real CSV and OFX exports, binds loopback as a non-root container, and has 85 tests at 91 percent coverage.

Next step: Treat an autonomous build the way you'd treat a junior pull request for a system that touches money: audit it adversarially and make every claim earn a test. The agent is great at producing something that looks shipped; the unglamorous correctness work is still yours. The build is cheap — the harden loop is the product.

Tags: #failure-modes #execution #systems

Placement Beats Rules

Dan Mercede — Sun, 28 Jun 2026 21:40:00 +0000

Placement Beats Rules

Short Essay · 2026-06-28 · 02:40 PM PT

Claim: Most agent-instruction files already encode the popular best-practice rules. The failures that remain are rarely a missing rule — they are a rule that never loads and a claim that was never verified.

A popular checklist of rules for agent-instruction files made the rounds again this week — think before coding, keep it simple, surgical changes, define success criteria, and a newer set about reproducing bugs with a test, debugging to root cause, and keeping a list of named failure modes the agent should self-check against.

The rules are good. But measure your own setup against them honestly and you may find the same thing I did: the file already encodes nine of ten. The gap isn't the rules. It's two things no rule list mentions.

**1. A failure-mode list only works if it's always loaded.** "Keep a list of named failure modes" is excellent advice that quietly assumes the list is in the context the model reads every turn. If yours lives in a separate notes store the agent has to search, it gets checked exactly when the agent thinks to search it — which is the opposite of when you need it. The recurring traps belong inline, in the always-loaded file, not one retrieval hop away.

**2. Context files truncate silently.** The always-loaded memory index I lean on was being injected *truncated* — its tail, which held a chunk of those "recall" lessons, was getting dropped above a size cap, and the harness only whispered it in a line that's easy to miss. Worse: a note *inside* that file confidently asserted the cap "isn't enforced," verified weeks earlier. Two failure modes stacked — a real limit, and a stale note swearing there wasn't one. The fix was small once seen: prove what the model actually received (it differs from what's on disk once you cross the cap), move the highest-value lessons to the top so the tail is what drops, and correct the note that lied.

The meta-lesson is the cheap one: a rule you adopt is only as good as its placement and your verification of it. Adopting the rule feels like progress. Confirming it loads, and re-checking the claims your own files make about themselves, is the part that actually moves the error rate.

Audit the two things the rule lists skip: is your recurring-failure list in always-loaded context or behind a retrieval step the agent only hits when it remembers to; and does your context file silently truncate above a size cap. Verify what the model actually loaded, keep the highest-value lessons at the top, and trust your own old notes least when they contradict live behavior.

Tags: #failure-modes #execution #systems

Self-Correcting Agents: Learning the Loop the Hard Way

Dan Mercede — Sat, 27 Jun 2026 17:12:52 +0000

I ran a multi-agent research agent over a hard question and it came back with a clean, confident verdict: "All 25 claims refuted by adversarial verification. Research inconclusive."

Every one of those 25 claims was true. Several cited real, recent papers I could pull up in another tab. Nothing had actually been checked.

The verifier hadn't disagreed with the research. It had crashed — every verification call hit transient API rate-limiting and returned nothing — and the loop scored "returned nothing" as "refuted." A fail-closed gate that cannot tell refuted by evidence apart from never adjudicated will lie to you with total confidence, and dress the lie up as diligence.

This guide is about that failure mode: why it happens, why it's the default outcome unless you design against it, how to spot it in your own loops, the gate that prevents it, and how to recover a run that has already produced a false "everything is false."

Who this is for: anyone building agent loops that check their own work — research agents, code agents with a review pass, anything with a "generate → verify → keep or discard" cycle. You don't need a specific framework. You need to take one idea seriously: silence is not a verdict.

1. The anatomy of a self-checking loop

Almost every serious agent loop has the same three roles, whatever you call them. The reasoning-frontiers survey of LLM reasoning fixes the vocabulary (arXiv:2504.09037):

A Reasoner generates candidate answers (the policy).
One or more Verifiers evaluate them (the judge / reward signal).
A Refiner revises based on verifier feedback.

All three can be the same model wearing different hats — that's exactly what "self-refinement" is. The research agent that lied to me was a fan-out version of this:

scope ──► search ──► fetch ──► extract claims ──► VERIFY (3-vote panel per claim) ──► synthesize
                                                      │
                                                      └─ each claim needs a quorum of votes to survive

The verify stage is the load-bearing one. It's where the loop decides what's true enough to keep. And it's the stage everyone gets wrong, because it's the stage where the failure is invisible: a broken Reasoner produces obvious garbage, but a broken Verifier produces clean, plausible, wrong verdicts.

2. The bug: a two-state gate in a three-state world

Here's the gate that shipped in the loop that failed me. Each claim got three independent verifier votes, tallied as notRefuted–refuted. The survive rule was reasonable on its face:

A claim survives if it has at least 2 valid votes and fewer than 2 refutations.

Read that rule carefully. It quietly assumes votes exist. When the verify phase crashed under rate-limiting, every vote came back empty. Each claim scored 0–0: zero refutations, but also zero valid votes. The survive rule said "fewer than 2 refutations? yes. At least 2 valid votes? no." — so the claim didn't survive.

So far, so correct. Refusing to pass an unverified claim is good design. The bug was one layer up, in the reporting: the claims that didn't survive were dumped into a bucket literally named refuted[], and the summary rendered that bucket as "all 25 claims refuted, research inconclusive."

The gate had collapsed a three-state world into two states:

The world has three outcomes, and you must keep them distinct:

Outcome	Meaning	What it tells you
CONFIRMED	quorum of valid votes, below the refute threshold	keep it
REFUTED	quorum of valid votes, at/above the refute threshold	drop it — the evidence says no
ABSTAINED	below quorum of valid votes (crash, timeout, null, rate-limit)	you don't know yet — re-run, escalate, or surface as pending

The one rule this whole guide rests on: an absence of votes is not a vote. Refuted is a verdict the evidence reached; abstained is a verdict the evidence never reached. A gate that conflates them converts every transient outage into a wall of false negatives — and because it fails closed, it looks responsible while doing it.

3. Why this is the default, not an edge case

You might think this is an exotic bug. It isn't — it's what you get unless you specifically prevent it, for three compounding reasons.

Fail-closed is the right instinct, and that's the trap. You should refuse to pass a claim you couldn't verify. Every safety-conscious engineer builds the gate to deny on uncertainty. The problem is that "deny" and "refute" land in the same bucket unless you force them apart. The safer your instinct, the more likely you are to mislabel silence as rejection.

Verifiers fail in correlated bursts. A single flaky vote is noise you'd shrug off. But verifier calls share infrastructure — the same API, the same rate limiter, the same network. When one abstains from a transient limit, they all do, at once. So you don't get one mislabeled claim; you get the whole run mislabeled in a single stroke. The blast radius is the entire output.

The summary layer launders the error. The gate's internal state might still be technically recoverable (0–0 is visibly different from 0–2). But by the time it reaches a one-line summary — "research inconclusive" — the distinction is gone. A human reading the summary has no way to tell a crash from a conclusion. The error becomes load-bearing for every downstream decision.

This is a specific, nasty instance of a general truth the research keeps surfacing: agents are far better at producing a good answer than at recognizing one. The verification step is the weakest link in the loop, so it deserves the most defensive design — not the least.

4. The research: verification is the bottleneck, not generation

Step back from the bug and look at why the verify stage is where loops live or die.

The verification gap. A 2026 benchmark of general LLM agents put a name to the failure (arXiv:2602.18998): when you sample several candidate answers, the right one is often already in your set — your pass@K is high, because the model generated it — but the model can't reliably select it, so the accuracy you actually realize lags far behind the answer you already have. Generation outruns judgment. This is the structural reason "just let the agent double-check itself" underdelivers, and the reason verifier quality dominates loop quality.

The self-correction blind spot. It gets sharper. Self-Correction Bench measured an average 64.5% blind-spot rate across 14 models (arXiv:2507.02778): models readily fix an error when it's pointed out in the prompt, but systematically fail to fix the same error in their own prior output. The likely cause is mundane — training data is overwhelmingly composed of clean, correct demonstrations, not error-then-correction traces, so the "notice and revise my own mistake" behavior is under-trained. The striking part: simply appending the word "Wait" cut blind spots by 89.3%. The capability is there; it just doesn't fire on its own.

Put those two findings together and the design conclusion is blunt: a loop that relies on a single model to silently self-judge is building on the part of the system that is measurably worst at the job. You cannot fix that with more self-reflection. You fix it with more, and more independent, verification — which is exactly why a verifier that quietly drops to zero is so destructive. It removes the one thing that was compensating for the model's blind spot, and it does so invisibly.

Verification is its own scaling axis. The frontier response to the verification gap is to scale the number of verifiers, not the cleverness of one. Multi-Agent Verification (arXiv:2502.20379) uses off-the-shelf models as independent "Aspect Verifiers" that vote — no training required — and shows weak-to-strong generalization: a panel of weaker verifiers can improve even a stronger generator. Rubric-grounded test-time verification (DeepVerifier, arXiv:2601.15808) extends the same idea to research agents, plug-and-play with no fine-tuning. The whole point of these systems is to make the verdict robust. A panel that silently abstains en masse is the precise opposite — it's a quorum that quietly dropped below quorum without telling anyone.

5. The fix: design the gate around three states

The fix is not subtle once you've named the problem. Make ABSTAINED a first-class state and never let it masquerade as either pass or fail.

Three rules implement it.

Rule 1 — classify on valid votes, not on net score. A vote only counts toward a verdict if the verifier actually ran and returned a parseable judgment. Before you tally notRefuted vs refuted, count how many votes are valid at all. If valid votes are below your quorum, the outcome is ABSTAINED, full stop — you never even reach the refute comparison.

for each claim:
    valid   = votes that actually ran and parsed
    refuted = valid votes that say "refuted"
    if len(valid) < QUORUM:                -> ABSTAINED   (not adjudicated)
    elif len(refuted) >= REFUTE_THRESHOLD: -> REFUTED     (evidence says no)
    else:                                  -> CONFIRMED    (keep)

For the 3-vote panel from §2, QUORUM = 2 and REFUTE_THRESHOLD = 2 — the very same thresholds as the broken gate. Nothing about the numbers changed; only the **order* of the checks and the keying on valid-vote count did.*

The single change from the broken gate is the first branch runs first and is keyed on valid-vote count, not on the refutation count. A 0–0 claim can never be reported as refuted because it exits at ABSTAINED before the refutation branch is ever evaluated.

Rule 2 — an error, timeout, or null is an abstention, and abstentions get retried, never counted clean. A verifier invocation that throws, times out, or returns nothing is a PENDING abstention — re-run it up to N times with backoff. Critically, an abstention is never silently treated as a pass or a fail. (This is the same discipline a good code-review gate needs: a reviewer subagent that errors out hasn't approved the diff — treating "the review crashed" as "the review is clean" is the same bug wearing a different shirt.)

Rule 3 — never transcribe "inconclusive" as a finding without checking whether the verifiers ran. This is the reporting-layer rule, and it's the one that actually saved me. Before a summary says "all refuted" or "inconclusive," it must inspect the shape of the votes that produced that verdict. If the run is dominated by 0–0 abstentions, the honest summary is "verification did not complete" — an operational failure to retry — not "the claims are false," a substantive finding. A crashed adjudicator's silence must never be rendered as a conclusion.

A compact way to enforce Rule 3 is a tiny, report-only reclassifier that you can run over any saved run output:

reclassify(run_output):
    for each claim in run_output.refuted_bucket:
        valid = parse_vote_tally(claim)
        if valid < QUORUM:  ->  ABSTAINED   (recoverable; the run never judged it)
        else:               ->  REFUTED     (a real verdict)
    if ABSTAINED dominates:  print "ABSTENTION-DOMINATED — the run did not refute anything"

(One naming note so the labels don't trip you: ABSTAINED is the terminal "not adjudicated" state — the same one a run's output later shows as "unverified"; PENDING is just its mid-retry sub-state. One concept, named once.) It changes nothing and runs nothing — it just re-reads the tallies and tells you whether your "all refuted" was a verdict or a crash. That single check is the difference between trusting a false negative and catching it.

6. Detecting it in a loop you already have

You can audit an existing loop for this failure without rebuilding it. The tells:

Symptom	What it usually means
A verify/review stage reports everything failed at once	correlated abstention (shared rate limit / outage), not a real mass rejection
Verdicts flip between runs with no input change	the gate is scoring transient failures as outcomes
The "failed" bucket has no per-item evidence (no refuting quote, no reason)	those items were never adjudicated — there's nothing to cite because nothing ran
A summary says "inconclusive" but the underlying items cite real, checkable sources	the loop discarded true results; verification, not the research, is what failed
Tightening a rate limit or shrinking a batch makes the "failure rate" drop	you're measuring infrastructure, not correctness

The deepest tell is the absence of evidence attached to a negative verdict. A genuine refutation can point at why — a contradicting source, a failing assertion, a counterexample. An abstention can't, because nothing happened. If your "no" can't show its work, it isn't a no.

7. Recovery: resume, don't re-run

Say it already happened — a run produced a false "everything refuted." The instinct is to run the whole expensive pipeline again from scratch — and in a real fan-out (mine spanned several stages and a couple of dozen sources), "from scratch" is a lot of wasted minutes and tokens. Don't. Most of that run is fine and recoverable. Recover in three steps, cheapest first.

Step 1 — reclassify (seconds, changes nothing). Run the report-only reclassifier from §5 over the saved output. It splits the "refuted" pile into genuinely refuted (a real quorum said no) versus abstained (below quorum — recoverable). If it comes back abstention-dominated, you now know the research itself is intact; only the verification failed. You've turned a false catastrophe into a known, bounded gap.

Step 2 — resume from the cached prefix; don't restart. A good loop harness caches completed stages. The scope, the search, the fetch, and every verifier vote that actually succeeded are still valid — re-running them wastes time and burns the same rate limit that caused the failure. Resume the run so that only the abstained verifier calls re-execute. You pay for the gap, not the whole pipeline.

Step 3 — top up the tail by hand, gently. For the handful of items still unverified after the resume, verify them sequentially — one source at a time, at a calm cadence. The shared rate limiter that caused the crash eases over time; a gentle, serial top-up slips under it where a parallel burst re-trips it. This is slow on purpose. Slow-and-complete beats fast-and-false.

A note on "future-dated" citations: when you re-verify, resist the urge to auto-reject a source just because it looks improbable — a paper with a publication date a few months out, say. In my run, several "suspicious" citations resolved to entirely real papers on inspection. Existence-check before you refute. That, too, is the difference between a verdict and a guess.

8. The broader principle: spend compute on generation and external verification

Zoom all the way out. The abstention bug is a sharp instance of a general lesson about where inference compute pays off in loops — and the evidence is increasingly clear that the popular instinct is backwards.

Brute repetition is the weak lever; verification is the binding constraint. Spending more compute by sampling several independent trajectories and selecting among them can help — the systematic study of test-time scaling for agents finds that scaling test-time compute, diversifying rollouts, and list-wise verification all move the needle (arXiv:2506.12928). But the payoff is far from automatic: a 2026 benchmark of general LLM agents found that neither parallel sampling nor longer sequential reasoning reliably improved performance — sequential scaling hit an "effective context ceiling," and parallel scaling was capped by the verification gap itself (arXiv:2602.18998). Read that twice: the thing throttling parallel sampling is the same selection problem this whole guide is about. The right answer is often already in your samples; the agent just can't pick it — and letting one agent simply "think longer" is the weakest lever of all.

Reflect selectively, not constantly. When you do refine, trigger it only when a step's score falls below a threshold, rather than reflecting at every step — knowing when to reflect beats reflecting often (arXiv:2506.12928). And self-verify-then-correct loops (SETS, arXiv:2501.19306) keep improving with compute where plain best-of-N plus majority vote saturates. The throughline: structured selectivity and stronger verification scale; brute repetition doesn't.

Now connect that back to the bug. If parallel generation puts the right answer in your candidate set, and the verification gap means selecting it is the hard part, then your verifier is the highest-leverage component in the entire loop. Which is exactly why a verifier that can silently fall to zero — and report that zero as a confident "no" — is the most expensive bug you can ship. It doesn't just add noise; it removes the one mechanism that was carrying the loop, and it hides the removal.

The design conclusion, stated as a single sentence: spend your inference budget on diverse generation plus stronger, independent, external verification — and build that verification so that when it fails, it says "I didn't check," never "the answer is no."

9. Checklist: an honest verification gate

Use this when you build or audit a self-checking loop.

[ ] Three states, always. Every verdict is CONFIRMED, REFUTED, or ABSTAINED. There is no fourth bucket, and ABSTAINED is never quietly merged into either of the others.
[ ] Classify on valid votes first. Count votes that actually ran and parsed before you compare refutations. Below quorum of valid votes ⇒ ABSTAINED, and you exit before the refute branch.
[ ] Errors/timeouts/nulls are abstentions. A verifier that throws, times out, or returns nothing is PENDING — retry up to N with backoff. It is never counted as clean and never counted as refuted.
[ ] A negative verdict must show evidence. A real REFUTED cites why (a contradicting source, a failing check). If a "no" has no attached evidence, treat it as an abstention until proven otherwise.
[ ] Guard the summary layer. Before any "inconclusive"/"all refuted" summary ships, inspect the vote shape. Abstention-dominated ⇒ report "verification did not complete," not "the claims are false."
[ ] Keep a report-only reclassifier. A tiny tool that re-reads a saved run's tallies and splits refuted from unverified turns a silent false-negative into a one-command catch.
[ ] Recover by resume, not restart. Cache completed stages and successful votes; re-run only the abstained calls. Top up the tail sequentially and gently so you don't re-trip the limiter.
[ ] Watch for correlated failure. If a whole batch "fails" at once, suspect shared infrastructure (rate limit, outage) before you believe a mass rejection.

The discipline behind every line is one sentence worth memorizing: a verifier's silence is missing data, not a "no." Build your loop so it can tell the difference, and it will fail honestly — which is the only kind of failure you can recover from.

Further reading: the systematic treatment of test-time scaling for agents (arXiv:2506.12928) and the 2026 benchmark of general agents that finds brute test-time scaling underdelivers — naming the context ceiling and the verification gap (arXiv:2602.18998); Multi-Agent Verification / Aspect Verifiers (arXiv:2502.20379) and rubric-guided test-time verification (arXiv:2601.15808) for verification as its own scaling axis; Self-Correction Bench for the 64.5% blind spot and the "Wait" activation (arXiv:2507.02778); SETS for self-verify-and-correct loops (arXiv:2501.19306); and the reasoning-frontiers survey for the Reasoner/Verifier/Refiner vocabulary (arXiv:2504.09037).

Originally published at danmercede.com.

Open-Source the Gate, Not the Verdict

Dan Mercede — Fri, 26 Jun 2026 20:08:19 +0000

It is tempting to think the intelligence is the product — that a sharper reviewer model is what makes AI code review safe. It is the opposite. The model that reads a diff is replaced within months. The thing that governs is the gate around it: the deterministic logic that default-denies on an unparseable or contradictory verdict and leaves a receipt of what it decided.

So the gate is what you open-source and the gate is what does the governing. The model sits behind a command seam and is swapped without touching the enforcement around it. Publishing the verdict ties governance to a model version; publishing the gate gives you something that holds as the models churn underneath it. The durable artifact is the boundary, not the brain.

Originally published at danmercede.com.