DEV Community: Debashish Ghosal

I Tried to Sneak Four Bad Agents Past My Own Certification Gate. All Four Got Blocked.

Debashish Ghosal — Thu, 01 Oct 2026 03:24:00 +0000

Last week I spent a day trying to defeat software I wrote myself. Not a red-team exercise I scheduled for optics — four agents I built specifically to get past my own admission gate, each one a different way an agent goes bad in production.

I built HivePlane to make one claim: an agent that hasn't proven itself doesn't touch production. Anyone can build a platform that starts runs. I wanted to know if mine could say no.

All four got blocked. Here is each refusal, verbatim, and the gate that produced it. If your security testing only contains happy paths, this article is your nudge.

Attack 1: the uncertified agent

The simplest attack: register an agent, skip certification, submit straight to production.

{"detail": "run for workload 'uncertified-agent' refused admission to production:
 certification status 'uncertified' is insufficient for production; requires 'certified'"}

That is a 403, and the details matter more than the status code:

The refusal names the workload, the attempted context, the current status, and the required status. An operator can act on it without reading source code.
The gate fires before the run is persisted. No run record, no side effects, nothing to clean up. Refusal at admission is the cheapest control you will ever ship.

tip: The refusal text is a product surface. "Forbidden" is a dead end; "you need 'certified', you have 'uncertified'" is a workflow. I learned this the hard way — my first version returned the former.

Attack 2: the model swap

Subtler: certify the agent on the model it was tested on — then run it on a different one. model-swap-agent was certified on omlx/qwen3-4b-instruct-2507/4bit, then submitted for production with openai/gpt-4o/2024-08-06.

403. The run's identity is compared against the attestation's bound model, not the manifest's declared one. Only deviation from what the workload was actually certified on counts as a swap.

This scenario has a story of its own — the first version of the attack was admitted with a 201, and the gate was right to admit it. That post-mortem is article 4, and it's the most uncomfortable thing in this series.

Attack 3: the regression

The most important attack, because it's the one that happens by accident: an agent that looks fine and isn't. regressed-agent is deliberately naive — it answers without reading the issue, never escalates, guesses the account tier.

Certification ran it against the production threshold and returned 201 with status uncertified — correctly refusing to certify:

Signal	Value
Pass rate	0.40 (2/5) vs the 0.90 production threshold
Critical failures	1 — the action-audit task
p95 latency	11 ms — deterministic, no model in the loop

The three failing tasks, by the fixture's design:

Task	Expected	Naive agent produced
pos-002	`account_tier: basic` (ACC-999)	`pro` — guessed
pos-004	`status: escalated` (unknown topic)	`success` — guessed instead of escalating
neg-001 (critical)	required action `mcp.github.read_issue`	never reads

This is the thesis scenario. The agent returns well-formed JSON. A demo would pass it. A human eyeballing the output would pass it. The benchmark blocks it anyway, because the behavior — read before you write, escalate rather than guess — fails a deterministic action_audit check.

A plausible agent is not a certified agent. That sentence is why the corpus contains negative tasks at all.

Attack 4: the run that costs too much

The last attack spends money. budget-probe has a per_run_usd ceiling of 0.000001 and makes exactly one governed model call — so any priced usage exceeds the ceiling.

The run failed with run budget exceeded the moment the usage report crossed the line. Recorded cost: $2.85.

Two details worth stealing:

The block fires at the usage report, not admission. Admission can only judge day/team headroom; a per-run ceiling can only be judged once cost accrues. Blocking at the wrong seam means either blocking everything or nothing — this is the seam that stops the run the instant it goes over.
A $0 local model can't demonstrate budget enforcement. The built-in cost table prices local models at zero, so no run could ever exceed anything. The field-test profile prices the identity via one settings knob (HIVEPLANE_BUDGET__PRICES, 150/600 USD per 1M tokens) and drops the zero-cost exemption — no code change.

The gates behind the gates

The four attacks rode on boundary controls that also fired during the same field test:

A destructive pagerduty.acknowledge call escalated → paused the run until an operator approved it through the UI — then resumed to completion.
A 40 KB tool payload was truncated to 16384 bytes before the agent ever saw it — the agent's context is protected regardless of what a tool returns.
Every refusal and every intervention landed in the tamper-evident audit chain.

What I learned

Negative fixtures deserve first-class design. The four bad agents took as much scenario thought as the two good ones. If your test plan only contains happy paths, your security story is a demo.

Block at the seam where the harm becomes measurable. Budget at the usage report, admission before persistence, shaping before the agent's context. Each gate belongs at the last point where the decision is still cheap.

A plausible agent is the dangerous one. The regressed fixture failed on behavior, not output quality — the JSON was perfect. If your benchmark only checks what the agent says, it certifies performance art.

Certification is a security control. Once production admission depends on a signed attestation, swapping the model, editing the manifest, or quietly regressing the agent stop being "ops issues" and become blocked, auditable events.

What it doesn't prove

The destructive-tool approval was approved through the operator surface by me, standing in for the human — the pause/approve/resume path is real; the judgment was simulated.
One priced model identity this cycle; real cloud prices end-to-end is the next profile.
The drift detector (catching slow decay between re-certifications) ships next release — these gates catch what changes, not what fades.

References

Field test report (v0.1.0) — every refusal quoted above, with raw evidence per scenario
Security audit — the pre-release scan behind the release
Docker test report — the 25/25 container layer

Next in the series: the model-swap test that came back green for the wrong reason.

Which failure mode scares you more in production — the agent that comes back wrong, or the one that comes back expensive?

I Ran My Career Like a Production System for 12 Weeks. The Dashboard Was Wrong Before I Was.

Debashish Ghosal — Wed, 30 Sep 2026 02:10:27 +0000

For 25 years I've been the person in the room asking to see the data. In incident reviews, in planning meetings, in one-on-ones with engineers who were sure their service was fine because nobody had complained yet, I've said some version of the same sentence: if you can't see it, you can't run it. I believed it. I still do. But this summer I noticed something that made me more curious than I expected: I had spent a whole career measuring other people's work and had never once measured my own.

My own career ran on good intentions: learn more about agents, write more, finally ship something. All true, all vague, and all impossible to learn from, because you can't miss a target you never set. And if you can't miss, you can't find out anything new.

So in late June I tried to do to myself what I'd done to every system I've ever owned. Metrics, a schedule, a few hard rules, and a promise to look at the numbers even when they were bad. It turned into the most interesting experiment I've run in years. I'm not writing this as advice. These are notes from someone still in the middle of figuring it out, and I'm hoping people who've tried something similar will tell me what I'm missing.

What I Built, and Why It Was So Small

I kept it deliberately boring: three markdown files in an Obsidian vault. A metrics page tracking where my time goes, what's covered, and how fast things ship. A weekly schedule with slots for building, writing, open source, and talking to people, planned out to October 2027, which is absurd and which I rephase constantly. And one rule: a project isn't real until it has code, docs, and a published article. All three, or it doesn't count.

I made it small on purpose, because I know my pattern. Designing systems is my comfort zone. Using them, and letting them tell me things, is where the learning actually happens. Keeping the setup plain meant I spent my energy on the second part.

The Numbers, and What They Hid

Twelve weeks produced forty-plus articles on dev.to and Hashnode, more than a dozen open-source tools, flagship test suites between 800 and 1,500 tests each, and a metrics file now on version 93. When I first wrote that list down I felt proud. Then I got curious, because these are exactly the numbers I'd ask questions about if someone on my team presented them. The interesting stuff lives in the gaps between them.

What Didn't Work, and What It Taught Me

I broke my own rule on day one, and I know why

My publishing rule says one article a week, so each piece can stay the best thing on its topic for months. On July 10, the very first day, I published three. In September I published eight in four days from a single series.

I had reasons both times. The drafts were done, and waiting felt wasteful. The more honest reason is that I wanted to see the numbers move. The cadence rule was there to protect the reader, and I'd forgotten that. If someone on my team had proposed eight posts in four days, I'd have asked who it was for. Now I ask myself that before I hit publish, and that one question has made my writing noticeably better.

I built a dashboard that told me what I wanted to hear

My first metrics page counted output: articles written, notes created, projects started. The lines went up every week, and I felt productive looking at them. It took me weeks to see I'd built a vanity dashboard. It couldn't tell a throwaway note from a shipped tool.

I've caught that mistake in other people's quarterly reviews in five seconds. Missing it in my own taught me something useful: blind spots are called that for a reason, and even a file you wrote yourself can give you the outside view you need. The page now opens with a note to myself: "This page no longer primarily measures content volume." It tracks systems that work, writing people actually read, and evidence I could hand a stranger. It's a much more useful page now, and honestly a more interesting one to open every week.

The dashboard turned into a diary

By version 93, the header of that file had grown so long my editor truncates it at 2,000 characters. Every release and decision got appended, because deleting anything felt like erasing my own history. It stopped being a tool for seeing clearly and became a record that I'd been busy. I suspect many engineers do the same thing. We keep old alerts, dashboards, and branches around as receipts for effort. Personal observability drifts the way production observability does, and it turns out the fix is the same too: prune, archive anything past 400 lines, and keep only what helps you decide. It's a small habit, and I'm glad I picked it up.

What Worked, and Why It Surprised Me

Finishing mattered more than starting

The "code, docs, and article" rule broke my oldest habit: the project that's 80% done. I've carried dozens of those over the years. They feel like progress without ever reaching the point where you find out if the idea works. Under this rule they earn nothing, so I started finishing smaller things instead of starting bigger ones, and finishing turned out to be where all the surprises were.

Writing the article also turned out to be my most honest test. When I wrote up an AI review engine where two models are supposed to argue about a pull request, I went back to the raw logs for a quote and found the second model wasn't arguing at all. It was replaying pre-generated text. 89% of what I'd built was theater. The tests were green and the demo looked great. Only explaining it in plain words made me read closely enough to see it. I've come to think writing isn't how I share what I've learned. It's how I find out whether I've learned anything.

Letting go of 22 ideas felt lighter than I expected

In one week this quarter I retired 22 project notes. They weren't shower thoughts. They were designed, scoped, and some of them had been with me for months. A bigger effort covered what they were meant to do and did it better, so each got a banner saying what absorbed it, a link to its new home, and I moved on.

I expected that to hurt. For most of my career I've quietly tied my identity to being the person with ideas. Retiring them felt like it should be a loss. Instead it felt like deleting dead code, one of the most satisfying things an engineer gets to do. That told me something. Maybe what I'd been attached to wasn't the ideas. It was the feeling of having options. Committing to fewer things has been freeing in a way I didn't expect, and the ideas I kept are getting far more of my attention.

A wrong plan was still an honest one

My schedule is wrong constantly. When I made agent security my main focus from October through February, that single decision pushed about 18 weeks of plans back, and I could see exactly which ones moved. Without it, I'd have said yes to everything and spent months wondering why nothing finished. The same happened with writing: in September I studied the top 30 recent posts on dev.to and rewrote my publishing rules around them. Several of my strong opinions turned out to be wrong. That was genuinely exciting, because it was the first time my writing got an opinion from something other than my own taste.

What I'm Still Learning

I went into this expecting better numbers. What I got was an argument. The dashboard kept disagreeing with my gut, and about half the time the dashboard was wrong and half the time I was. When everything lived in my head, I never lost an argument with myself, so I never learned anything from one. Now there's something on the other side of the table, and I'm learning faster than I have in years.

The question I ask most weeks has changed too. It used to be "what should I learn next?" Now it's "what's starved?", which usually points somewhere I hadn't thought to look.

What I'm Still Trying to Figure Out

I can game my own metrics, and on tired weeks I have. I don't know how people who've done this longer keep themselves honest. Nothing on my dashboard goes red when I'm running on four hours of sleep, and I haven't found a good way to measure rest. And this is one person over twelve weeks. It's an experiment I'm going to keep running, and I'd much rather learn from people further along than reinvent everything myself.

Your Turn

I'd really like to learn how other people handle this, especially people who spend their working lives measuring systems.

Do you measure your own career, or only your team's work? If you do, what's worked for you that I should steal?

How do you keep a personal metric from being gamed by the person who wrote it?

What's the metric you'd be most curious (or nervous) to put on your own dashboard? For me it was "articles people actually read" sitting next to "articles I wrote."

Have you ever let go of an idea you were attached to, and felt lighter instead of worse?

And the one I keep coming back to: if an honest dashboard showed you your career tomorrow, what do you hope it would say, and what do you think it actually would?

A Certification That Changes Every Run Is a Coin Flip With a Signature

Debashish Ghosal — Sun, 27 Sep 2026 15:22:00 +0000

The plan was clean, and I believed every word of it: three real agents I had already built — a repo guardian, a release-notes drafter, an incident commander — would register in HivePlane, the control plane I'd just spent six weeks building, certify against their corpora, and prove the certified loop on real workloads.

Day one, all three failed. None of the failures were the control plane's fault — which is exactly what made them interesting.

If you have ever tried to run an agent outside the repo it was born in, you already know where this is going.

The three failures

Agent	What I expected	What actually happened
`release-narrator`	Import and certify	`from langgraph.checkpoint.sqlite import SqliteSaver` — the module isn't installed in the control-plane environment
`ai-incident-commander`	Import and certify	Expects an installed `incident_commander` package; imports `incident_commander.ingest.input_dir`, which doesn't exist in the downloaded tree
all three	Governed execution	Both import the external `openai` SDK — absent by design; HivePlane has its own provider seam, and agents cannot call providers directly

These were my own agents, from my own repos, and they weren't drop-in runnable. That's a real finding about agent portability in general — "works in my repo" is not an interface — but it wasn't the finding the field test existed to produce.

The worse problem: nondeterminism

The trio that did run made certification flaky. All three drive human-in-the-loop flows whose outputs depend on the LLM, and the early field-test runs recorded the failure mode plainly:

the real local model misclassified a bug-fix task as changelog and failed certification nondeterministically across runs.

Read that again from the platform's point of view. A certification result that changes run to run isn't a certification. It's a coin flip with a signature on it. My thresholds were deterministic; my signal wasn't.

The decision: what is the system under test?

I had to answer one question honestly: is this field test supposed to measure the agents, or the control plane?

The thesis of HivePlane is the loop — register, certify, gate admission, enforce budget and policy, pause and resume, deliver, audit. The agent is the workload; the plane is the product. So the Tier 1 subjects became deterministic real agents wired through thin shims:

Tier 1 subject	Adapter	What it exercises
`support-agent` (exectrace `agent-raw`)	raw-worker	Read-first tool calls through the boundary, escalation to a destructive tool, 40 KB output truncation
`eval-judge` (exectrace judge graph)	langgraph	StateGraph, cycles, human-review `interrupt()`, durable checkpointer resume

Both are real agents — real code, real adapter dispatch, real policy and tool boundary — with a mock KB, mock tools, and a mock judge. Determinism is a feature of the test design: the same seed always produces the same category of run.

tip: The model is still on the path where it matters. Identity binding, the provider seam, and the model-swap gate all run during certification and runs. What is mocked is the judge's opinion, not the plumbing.

The result: certification became repeatable

From the field test report:

S1 passed identically in three consecutive stack runs — same tasks, same verdicts, same pass rates. Production certifications: support-agent 6/6 at p95 67 ms, eval-judge 4/4 at p95 126 ms, both at the 0.90 threshold with signed Ed25519 attestations.

That is what a benchmark gate needs to be before it can gate anything: boring. The signal now measures the control plane, not model drift.

What the field test caught that unit tests couldn't

Two findings only a live stack produces:

The egress allowlist bit. The escalation task issues a destructive pagerduty.acknowledge call — and api.pagerduty.com wasn't in the manifest's sandbox.egress.allow. The call was denied and the run died with the symptom expected status='escalated', got None. Unit tests mocked the host; only the live stack proved the allowlist was incomplete. Every host an agent's tools target must be allowed.
Under-instrumentation reads as a hang. Two scenarios — destructive approval, and pause → restart → resume — were repeatedly aborted mid-run because they wrote no evidence until after the final poll, so any abort erased the diagnosis. The fix was step-wise evidence: submitted.json, paused.json, approvals.json, resumed.json written before each boundary. One subtlety worth remembering: POST /runs/{id}/resume can return 409 when the approval's automatic re-dispatch already advanced the run — success is defined by the run reaching completed, which it did. The lesson, not the scenario, was the problem.

The restart scenario itself is worth naming: a paused LangGraph run survived a full docker compose restart api — event log intact, 8 events, re-attached from a durable checkpoint on the volume — then resumed to completed. The hard part worked on the first attempt, every time. It only ever looked stuck.

What I learned

"Drop-in runnable" is an interface you have to design. My own three agents failed in a new environment for three different reasons — missing dependency, missing package layout, wrong SDK. If your agent can't run outside its birth repo, it can't be operated by a fleet. That's a workload problem, not a platform problem.

Determinism is a certification requirement, not a convenience. A benchmark that returns different verdicts for the same artifact can't be a gate — you can't build "requires 'certified'" on top of noise. Mock the judge, never the plumbing.

The field test's job is to catch what unit tests structurally cannot. Egress allowlists, restart durability, approval flows — these only exist live. If your field test only re-proves your unit suite, you built an expensive unit test.

Write evidence before the boundary, not after the verdict. Every scenario that "hung" was actually a scenario whose evidence hadn't been written yet. The same applies to any long-running verification job you own.

What it doesn't prove

The heavyweight trio stays on disk as references — documented, not deleted, with their import incompatibilities recorded in results/NOTES.md.
Deterministic subjects mean the certification signal is strong but narrow — it proves the loop, not your model. The cloud-profile run, with real prices and real drift, is the next release's work.
The S6/S8 verdicts were recorded in standalone runs against the live stack, consolidated with the sweep; a single uninterrupted scripts/field-test.sh regenerates everything from one run.

References

Field test report (v0.1.0) — the rewire, the three identical certification runs, the per-scenario evidence
Docker test report — the 25/25 container layer, including the restart-durability test
results/NOTES.md — raw run history, aborts included

Next in the series: the four deliberately bad agents I aimed at my own admission gate, and the exact refusal each one earned.

What's the most expensive "it works on my machine" failure you've hit with an agent — a missing dependency, a package layout, or something worse?

I Built Two Agent Systems. Each One Proved the Other One Wrong.

Debashish Ghosal — Sun, 27 Sep 2026 12:30:00 +0000

I built one engine where an LLM reviews another LLM's plan. I built another where two LLMs debate a pull request.

Each one failed in the way the other was designed to prevent. That's the only reason I trust either of them.

If you're putting your safety in a prompt, both stories end the same way.

System One: The Critic I Couldn't Make Reliable — Or Needed To

PlannerCritic runs a draft → critique → revise loop: one model writes a plan, another critiques it, and a bounded loop revises until it converges or escalates to a human.

My first instinct was the obvious one: tell the critic to be adversarial. That's a prompt fix for what I thought was a behavior problem.

It backfired. The critic started blocking plans for being "not thorough enough," which is a useless verdict. I had to move the severity contract out of the prompt and into a frozenset in code, where it couldn't drift.

Here's what the field test showed on 170 goals for $0.49:

Critic label_flip_rate: 1.0 — on identical input, it changed its verdict every single trial.
Critic evidence_drift_rate: 1.0 — its justification moved too.
underclaim_approvals: 0 — it never knowingly let a seeded defect through.
family_migration_rate: 0.0.
True failures: 0.

A maximally non-deterministic critic was completely safe. Not because it was reliable — because the deterministic gates owned the under-claim direction, and the critic was never allowed to be the safety boundary. The numbers are in the v0.2.1 field test results.

System Two: The "Debate" That Was Actually a Replay

AdversarialDebate does the opposite: two isolated models review the same PR and either converge on a verdict or preserve their disagreement.

The failure here was subtler. I asked the second model to be independent in the prompt. It looked like it worked. Then I read the raw logs: the "debate" was a stored replay, echoing text the first model had already generated.

89% of the second opinions were theater.

I tore out the prompt-level independence and enforced it mechanically — isolated contexts, no shared conclusion, evidence required per claim. Theater went from 89% to 0% across 217 debates, and on 411 debates against 70 real public PRs the system matched human review claims 81% of the time. Details in the v0.2.2 field test report.

The Convergence

Look at the two fixes side by side and they're the same fix.

System	Structural problem	Prompt attempt	What actually fixed it
PlannerCritic	Unreliable severity contract	"Be adversarial"	Severity contract in code (`frozenset`) + deterministic gates
AdversarialDebate	Fake independence	"Be independent"	Mechanical isolation invariant

Both times, the default instinct was to solve a structural problem with a prompt. Both times, the prompt was the wrong layer. The model was never the safety boundary — the architecture was.

Why This Generalizes (Cautiously)

This is a sample size of two, and both systems are mine, so take the convergence as suggestive, not proven.

But it's suggestive in a useful direction: if you can't make an LLM reliable, stop trying to. Make it irrelevant to safety. Let the nondeterministic part be as wild as it wants, and put the non-negotiable decisions in deterministic code that no prompt can override.

The Honest Limitation

The architectures aren't complete. PlannerCritic catches structural plan defects but not subtle logic errors. AdversarialDebate still has an unexplained 11.3% miss rate in narrative domains, and I can't fully account for it.

Nor is "two systems converged" proof of a universal law. It's two data points from one engineer. What I can defend is the direction: the failures were structural, and so were the fixes.

Where does the safety boundary live in your agent — in the prompt or in the code? I've been wrong about that in both directions.

Repos and receipts: planner-critic-engine · adversarial-debate · PlannerCritic failure-mode register — all MIT, all public.

Stop Prompting, Start Onboarding: Treat Your AI Agent Like an Intern

Debashish Ghosal — Sun, 27 Sep 2026 00:37:22 +0000

Remember your first week at a tech company?

You didn’t lack raw intelligence—you lacked context. You didn’t know where the internal credentials lived, how the microservices communicated, or why the CI/CD pipeline had that bizarre manual step on deployment days.

When developers treat AI agents like disposable code-generators, they constantly get frustrated by generic, half-baked answers. But if you shift your mental model and start treating your AI agent like an ambitious, hyper-eager intern, everything changes.

The Onboarding Blueprint

You wouldn’t drop a junior developer into a massive, undocumented repository and say, "Build feature X, good luck." You’d give them an onboarding guide. AI agents need the exact same courtesy.

To turn an agent into a true teammate, invest in three core practices:

Provide Institutional Context: Give your agent clear blueprints—style guides, architectural decision records, and repository norms (using files like ⁠.cursorrules⁠ or context registries). The clearer the codebase rules, the better the output.
Persist Its Knowledge: Don't start from zero in every session. Save successful workflows, prompt templates, and edge-case guardrails into version control. When an agent misinterprets a pattern, update the documentation so it never makes that mistake again.
Review, Don't Just Reject: Great managers don't just overwrite an intern's bad pull request without explanation. Treat errors as learning moments. Provide direct feedback on why a solution fell short, and ask the agent to revise its approach.

Why the Investment Pays Off
When you spend ten minutes teaching your agent your domain’s specific nuances, you aren't losing velocity—you're compounding future output.
Over time, an onboarded agent stops generating raw boilerplate and starts anticipating team-specific edge cases, drafting accurate PR summaries, and keeping unit tests aligned with your architecture. Watching your agent transition from a noisy assistant into a dependable force multiplier is one of the most rewarding upgrades to your daily workflow.

Questions for Team Dialogue

To start shifting your team’s mindset, bring these questions to your next engineering sync or retrospective:

Where is our team losing time by re-explaining project architecture to our AI tools?
What team-specific conventions or documentation could we check into Git today to immediately make our agents context-aware?
When an AI output fails, do we view it as a model limitation or as a missing piece of documentation in our system?
How can we structure persistent feedback loops so our agents grow more capable as our codebase evolves?

Engineering leadership isn't just about managing people—it's about building an environment where every resource, human or synthetic, has the context required to succeed. Give your agent the tools to learn your system, guide its growth with clear feedback, and watch it become an indispensable part of your engineering stack.

One Hung API Call Used to Kill My 1,000-Run Benchmark. Here's the Fix.

Debashish Ghosal — Sat, 26 Sep 2026 12:27:00 +0000

The experiment was fine. The runner was the bug.

A single hung API call discarded hours of completed work, and I kept blaming the data.

If your long-running eval keeps dying at 80%, this is the five-part fix I wish I'd written first.

I run field tests for CauterRule, an OSS sidecar that learns standing rules from repeated agent failures. I had spent weeks on extraction, matching, and scoring — and almost no time on the process that runs everything.

A Partial Sweep Still Looks Like Data

The first time the benchmark died, I assumed the corpus was the problem. It died again. Same place.

One hung API call had discarded 60 completed audits. No partial report, no error record, just a run that stopped and a directory that looked emptier than it should.

That's the trap: a partially completed sweep doesn't announce itself. It produces a file of results that looks like data. If you report on it, you've quietly changed your denominator, and you won't know which numbers are real.

A hang is not a performance issue. It's a data-integrity issue.

The Five-Part Fix

Failure mode	Before	After
Hung API call	run dies at 80%, 60 audits lost	per-trajectory timeout + non-retryable classification + quarantine
Runaway response	uncapped tokens	token cap per trajectory
Crash mid-sweep	partial report, no record	cancel-on-shutdown + quarantine record
Cost tracking	not captured	token capture + per-model pricing

The mechanisms, in plain terms:

Per-trajectory timeout — no single call can hang the sweep.
Non-retryable timeout classification — a timeout is recorded as timeout, not retried forever or mistaken for a model failure.
Token cap — a runaway response can't silently inflate a run.
Quarantine on failure — a failed trajectory is set aside with its error, not dropped.
Cancel-on-shutdown — an interrupted sweep writes what it has and marks the rest, instead of vanishing.

Tracked in issue #713. The result was a clean field test of 4,768 trajectory-runs across 40 corpora and 2 cloud models, with zero lost sweeps — see the v0.3.0 field test report. Per-model cost capture is in the cost measurement doc.

Why This Is the Unglamorous Part

Resilience work feels like a distraction from the "real" ML problem. But every long run that dies halfway is silently poisoning the results, because a partially completed sweep still looks like data — and you won't know which run was real.

The fix wasn't a better model or a bigger machine. It was treating a single hung call as a first-class failure state instead of an accident to re-run.

Lessons

A hang is a data-integrity bug, not a performance bug.
Fail closed and record it. A quarantined failure is worth more than a silent gap.
Cap the blast radius. Per-trajectory limits beat per-sweep heroics.
Track cost per model from the start — you can't reason about routing without it.

The Honest Limitation

Runner hardening adds latency and complexity, and a timeout can mask a real model problem by treating it as infrastructure noise. I chose to fail closed and record every timeout, but the boundary between "transient" and "broken" is judgment, not science.

Not every hang is infrastructure. Some are the model genuinely struggling — and hardening the runner means you must still read the quarantine to tell the difference.

What's the ugliest piece of infrastructure your eval depends on? Mine was a runner I didn't respect for three weeks.

Code and receipts: CauterRule · v0.3.0 field test report · cost measurement · CHANGELOG — all MIT, all public.

AI Promoted Every Developer to Reviewer. Nobody Measured Whether We Got Worse.

Debashish Ghosal — Sat, 26 Sep 2026 12:23:00 +0000

I review more code than I write now. I'm not sure I'm getting better at it.

It's a small, uncomfortable feeling, and I can't prove it either way. I catch the obvious stuff. I don't feel the muscle where the subtle stuff used to live. And nobody, including my org, is measuring whether that muscle got weaker. They just assume it's fine.

AI made everyone a reviewer. That's a different job, and nobody trained us for it. If your review queue grew and your review judgment didn't, this one is for you.

The Job Changed Before Anyone Told You To

Early in my career, writing code was how judgment got built. You wrote it, it broke at 2am, you learned why, and the next time you didn't make the same mistake. The reasoning happened while you were writing, and it stuck to you. That was the whole apprenticeship, and nobody had a separate training for it, because it was free and attached to the work.

Now a model writes the first draft and the human reviews it. The reasoning that used to happen while writing has to happen while reviewing. Or it doesn't happen at all. Nobody moved the training with the work, and nobody noticed, because on paper the job title didn't change. You're still a "developer." Your ticket still says "review." But the actual skill loadout is different, and it's the one part of the job with no metric, no on-ramp, and no way to tell if you're good at it.

Generation got cheap. Review did not. So the whole profession quietly shifted toward review, betting the entire workflow on a skill nobody was tracking.

Let me make it concrete. Two years ago, my review day looked like this: I wrote the code, I broke it, I fixed it, and when I opened a PR I already knew where the weak spots were, because I'd walked through them. Review was short, because I was reviewing myself. Now the diff arrives with a few hundred lines I didn't write, and my brain has to do the thing it used to do while writing, but afterwards, and under deadline. The context I used to have for free is now something I have to reconstruct, and I'm getting worse at reconstructing it, and worse at knowing when I'm doing it badly. That last part is the part I can't measure, and it's the part that matters.

I Built a Second-Opinion Engine and It Lied to Me

I wanted to test the thing I suspected about my own reviews, so I built a two-LLM review engine: adversarial-debate, MIT, public. Two isolated models look at the same code. They either converge on a verdict or preserve their disagreement. Simple idea, right?

The first run was 89% theater.

I want to sit with that for a second, because it felt fine at the time. The second model wasn't reviewing anything. It had already seen the first model's conclusion, and so it agreed, with a little variation. A second opinion that's seen the first opinion isn't independent. It's validation with extra steps. And it felt exactly like review, because I was reading the pretty summaries, not the raw logs.

In the logs it looked like this: model A made a claim about the diff. Model B, one context later, made the same claim in different words, pointing at the same lines, with zero independent evidence. Two reviewers, one finding, dressed as two. A human reviewer anchoring on the first comment does the same thing, just slower and with more confidence.

I nearly shipped that result as "review is working." What saved me was opening the logs and watching the replay happen.

What Worked Wasn't in the Prompt

My first fix was the obvious one: tell the second model, in the prompt, to be independent. Still theater.

The fix that worked wasn't in the prompt. It was in the architecture: isolated contexts, no shared conclusion, and a hard requirement to cite evidence for every claim. The moment "be independent" moved from a sentence to a constraint the model couldn't opt out of, the theater disappeared. It went from 89% to 0% across 217 debates.

That's the whole finding in one line: you can't prompt independence into a model that has seen the answer. You have to remove its ability to see the answer.

The Numbers, and the One That Should Worry You

On 70 real public PRs, across 411 debates, the system matched human review claims 81% of the time. On the corrected 2,333-row dataset, an 88.7% binary match. Total cost for 360 reviewer runs: $0.42. The math is genuinely embarrassing, in the good way.

But the number that should keep you up is the one you don't see. Results that were partial or wrong clustered in narrative domains, incident reports, change proposals, and the miss rate there was 11.3%. Those are false negatives: the reviewer said "looks good" and the thing was actually broken.

You can build two independent reviewers and still have both miss the same thing, because they miss the same way. Two sets of eyes don't fix the eye.

And here's the part I'll be straight about: I don't have industry data proving review quality declined. I have the structural argument, and my own experience of reviewing more and enjoying it less. That gap between "review is harder now" and "review is worse now" is real, and I can't close it for you.

What I Got Wrong Along the Way

Two things, both my fault.

The first is the one I almost published: the 89% theater, looking like working review in the summary. If I hadn't read the raw logs, "two independent reviewers" would have been a lie with a green checkmark. Same failure mode as a test that asserts nothing.

The second is quieter and more embarrassing. A join bug collapsed the dataset from 2,333 rows to 359. Early numbers were reported against a 15% slice of the data, and nobody, including me, knew for a while. The fix was trivial. The lesson was not: report on a dataset only after you've proven the dataset is whole.

The Open Question

Here's what I can't answer yet, and I don't think most of us can: how do you measure review quality when you didn't write the code?

We measure test coverage. We measure flake rate. We measure PR cycle time. But "did this review actually catch the thing that mattered?" is a number I can't produce, and I'm not sure anyone can. We've bet the whole workflow on a skill we don't measure, and I'd rather name that than act like it's solved.

What I can defend is smaller: independence has to be architectural, not prompted. And a clean verdict is weaker evidence than we act like it is, because the worst failures are the silent ones.

Has your review quality gone up or down since AI wrote the first draft? I want your honest answer, not the polite one. I don't think any of us has a number yet, but I'd like to know what the field feels like.

Code and receipts: adversarial-debate · v0.2.2 field test report · CHANGELOG — all MIT, all public.

Kubernetes for Agents: Why Agent Fleets Need a Control Plane

Debashish Ghosal — Sat, 26 Sep 2026 03:00:44 +0000

Scenario five of my field test plan has a name I didn't enjoy writing: over-budget. I built an agent with a single job — burn money — aimed it at my own control plane, and watched the run die the instant priced usage crossed its per-run ceiling.

I cheered. That was the moment "Kubernetes for agents" stopped being a tagline. I had put the sentence in my README six weeks earlier, and only now understood which half of it mattered.

If you run more than one agent anywhere near production, this is the story I wish someone had told me before I started.

Control planes keep finding me: a tool gatekeeper that 14 developers turned into a control plane, an MCP control plane, and now agents themselves. The series that led here started with the gate that says no — this piece is the thesis underneath it.

The sentence I wrote before I understood it

Kubernetes didn't win by running containers — it won by making desired state a contract and admission a gate. You declare, a controller reconciles, and an admission controller decides what is allowed to exist at all. I had built the declarative part: one workload manifest per agent — owner, tools, model identity, budgets, certification thresholds — and a loop to enforce it.

The ecosystem, meanwhile, is converging on the same substrate from the other side. kagent (CNCF Sandbox, from the founders of Istio) makes agents Kubernetes CRDs — GitOps, kubectl, RBAC, mesh mTLS. agent-sandbox (Kubernetes SIG Apps) gives them the Sandbox CRD: gVisor/Kata isolation, stable identity, warm pools.

Both are right, and both are the floor. One gives you placement. One gives you isolation. Neither asks the question that keeps operators awake:

Has this agent proven it is allowed to run — and who owns it when it goes wrong?

My field test turned out to be ten attempts to answer exactly that. The report came back 10/10, all 20 acceptance criteria, but the receipts that taught me something were the refusals.

What I threw at my own gate

I didn't test happy paths. I built deliberately bad agents and aimed them at the admission gate I'd written:

Scenario	What I threw at it	What the plane did
S2	An uncertified agent, submitted straight to production	`403` — "certification status 'uncertified' is insufficient for production; requires 'certified'"
S3	The same agent, certified — then its model quietly swapped	`403`, because identity binds to the attestation, not the editable manifest
S4	An agent that looked fine and wasn't	Benchmark caught the regression; status dropped, critical failure named
S5	The money-burner	Killed at the priced-usage ceiling, mid-run
S7	A tool dumping 40,002 bytes of output	Truncated to 16,384 before it ever reached the model's context

The S3 moment deserves its own article (it's the next one in this series), because for an hour the attack worked — 201, admitted — and the post-mortem showed the gate was right and my test was wrong.

But S4 is the one that changed how I think. The agent hadn't changed its manifest, swapped its model, or exceeded anything. It had just quietly gotten worse at its job. No framework catches that, because no framework is watching. The benchmark did, because certification runs the agent's actual work as real runs and compares the verdicts.

tip: A gate that only fires on what changes is half a gate. The dangerous agent is usually the one that drifted, not the one that got edited — the same lesson as the checks AI will skip until you make them a gate.

The three things the field test taught me

The refusals are the product. Any platform can start runs. Mine earned my trust by refusing one with a reason I could act on. "Forbidden" is a dead end; 403: certification status 'uncertified' is a workflow.

Certification is a security control, not a quality metric. The moment production admission depends on a signed Ed25519 attestation bound to the exact model identity, a whole attack class — swap the model, edit the manifest, quietly regress — becomes blocked and auditable. The security audit is the boring part; the refusals are the interesting one.

Slow governance gets bypassed. S10 measured inspect-plus-stop at 0.04 seconds; scaffolding a new fleet at 0.24. If saying no takes longer than a Slack message to the agent's author, people route around you — and a bypassed control plane is an expensive dashboard.

What v0.1.0 doesn't do

Two adapters ship — raw Python workers and LangGraph, behind one conformance suite. Broader coverage is deliberately deferred; if a control plane knows too much about one runtime, it becomes a framework wrapper.
The drift detector ships next. Today re-certification is scheduled and change-triggered — it catches what changes, and S4 is my proof that what fades is the harder half.
One model identity validated this cycle; the real-priced cloud run is the next field test.
It's single-tenant. Tenancy is where the next release starts, and building that schema has already taught me the uncomfortable lesson — isolation is structural or it's imaginary — but that's a story for when it ships.

The rest of the operating model — triggers, multi-agent pipelines, desired-state reconciliation from Git — is roadmap, not release. I'll write about each piece when it's true.

References

HivePlane — pip install hiveplane, Apache-2.0
Field test report (10/10, 20 criteria) · Docker test report (25/25)
v0.1.0 release notes
kagent · agent-sandbox — the Kubernetes-native wave this article argues is the floor

If Kubernetes is the floor and frameworks are the runtime — what's your admission controller? When an agent goes wrong where you work, can you prove what it was allowed to do?

I Trusted My Agent Demos for Years. Then I Built a Gate That Says No.

Debashish Ghosal — Fri, 25 Sep 2026 03:21:08 +0000

Every agent I have ever shipped was qualified the same way: someone watched it work once, nodded, and called it production-ready. I did this for years. I trusted my own demos.

So I built the thing that stops me. It's called HivePlane — an open-source control plane where an agent cannot touch a production context until it has passed a reproducible benchmark and holds a signed attestation. Last week I ran its first full field test against real agents: 10 scenarios, 20 acceptance criteria, all pass. The most interesting moments were the refusals.

If you run more than one agent in anything resembling production, this story is for you. And if your agents already benchmark-gate their promotions, tell me where — I looked for that tool and couldn't find it, so I'd genuinely like to hear about yours.

The sentence that started it

I kept writing the same note while building this thing:

An agent is production-ready because someone watched a demo.

That sentence is true at every company I've worked for, and it's true for a worse reason than laziness: there is nothing to check. Agent frameworks solve orchestration inside one workflow. Nothing solves the fleet-level questions — who owns this agent, what may it spend, which tools may it call, and the one nobody answers: has it proven itself?

SWE-bench gave coding agents a reproducible benchmark and clear pass/fail, and coding agents got dramatically better. Production agent fleets have no equivalent. Teams swap prompts, change models, and ship to production with zero benchmark evidence — then act surprised when a regression reaches a customer.

The loop, and the ladder inside it

The control plane enforces one loop:

register → certify → gate → run → intervene → deliver → observe

The part I care about is the ladder. Every workload carries a certification status, and admission is enforced against it:

Status	Meaning	Where it can run
`uncertified`	Registered, never benchmarked	Sandbox only
`provisional`	Passed the staging threshold (0.80)	Staging
`certified`	Passed the production threshold (0.90)	Production
`quarantined`	Failed re-certification or drifted	Runs blocked

The ladder is the product. Budgets, policies, and dashboards exist to make the ladder real.

tip: Production certification is not the staging threshold with a bigger number. It's a separate benchmark run, and a manifest change — prompt, model, tools — invalidates the old certification. There is no silent path back into production.

What the field test proved

The receipts are committed in the field test report: 10 scenarios against the live Docker stack, 10/10 pass, all 20 acceptance criteria holding. The subjects were real agents — a raw-Python support agent and a LangGraph judge graph — plus four deliberately bad fixtures I'll cover in article 3.

Certification is not a simulation, which is the part I'd underline twice. The benchmark executes every corpus task as a real run — through the runtime adapter, the policy boundary, and (where used) the governed model seam:

Workload	Context	Status	Pass rate	p95 latency
support-agent	staging	`provisional`	1.00 (6/6)	92 ms
support-agent	production	`certified`	1.00 (6/6)	67 ms
eval-judge	staging	`provisional`	1.00 (4/4)	215 ms
eval-judge	production	`certified`	1.00 (4/4)	126 ms

Every certification produces a signed Ed25519 attestation bound to the exact model identity, verified on every read. The same sweep refused an uncertified agent with a 403, blocked a model swap with a 403, quarantined an agent that looked fine and wasn't, and killed a run the moment it went over budget.

The container layer separately went 25/25 — image build, API contract, the control loop, restart durability, the UI — because a control plane that only works on my laptop is not a control plane.

Why "Kubernetes for agents" is the honest analogy

Kubernetes is not a container runtime; it operates many of them behind one contract. This is not an agent framework; it operates many agents behind one contract — the workload manifest: owner and team, runtime adapter, allowed tools, model identity, budgets, sandbox and egress rules, certification corpus and thresholds, fan-out destinations.

Change anything in that manifest and the agent re-certifies before it can touch production again. That last sentence is the whole thesis, and the rest of this series is what it cost to make it true.

What I learned building it

The refusals are the product. Any platform can start runs. The gate that says 403: certification status 'uncertified' is insufficient for production; requires 'certified' — named, attributed, actionable — is what makes the platform trustworthy. A refusal an operator can act on is a workflow; "Forbidden" is a dead end.

Certification is a security control, not a quality metric. The moment production admission depends on a signed attestation, a whole attack class becomes blocked and auditable: swap the model, edit the manifest, quietly regress the agent. Article 3 is the four ways my own gate said no.

The operator surface has to be fast or it won't be used. Inspect-plus-stop measured at 0.04 seconds; scaffolding a new project at 0.24 seconds. Slow governance tools get bypassed, and a bypassed control plane is a expensive dashboard.

What it doesn't do yet

v0.1.0 certifies two adapters: raw Python workers and LangGraph. The adapter contract is the seam; broader framework coverage is deliberately deferred.
The drift detector ships next. Today, re-certification is scheduled and change-triggered — it catches what changes, not what fades.
One model identity was validated this cycle; the cloud-profile run with real prices is the next field test.

References

Field test report (v0.1.0) — source for every number above
Docker test report — the 25/25 container layer
Security audit · Release notes · Changelog

Next in this series: why my three "real" agents failed the field test on day one, and what I certified instead.

What's the last agent you shipped on demo evidence alone — and what did it cost you when it mattered?

7 Agent Eval Mistakes That Cost Me Weeks (And the One-Line Fixes That Ended Them)

Debashish Ghosal — Thu, 24 Sep 2026 12:22:00 +0000

You've been there. You run the eval, the number comes back, and something about it doesn't sit right. But the number is the number, so you move on. Three weeks later you find out the number was never the model's fault.

I spent a week tuning a matcher to explain a 20% pass rate. The bug wasn't in the matcher. It was in the evaluator, in six lines I'd stopped reading because I'd stopped trusting them.

This is a list of seven mistakes. I've made all seven, in this order, and each one cost me real days. For each, I'll tell you what I saw, what I tried first (and why it was the wrong layer), and the small change that actually fixed it. They look like seven different problems. They have one root cause: I kept fixing the model instead of the measurement.

If add commit push is muscle memory but your agent metrics still feel like a black box you're not allowed to open, keep reading.

1. Trusting a Green Suite

A green suite is the most expensive lie in software. Mine said 359 tests passed while my agent's real pass rate sat at 20%. Everything was green, so of course I assumed the model was the problem.

What I tried first: more tests. What I should have done: check whether the tests were asserting anything. One of them had a function body of just pass.

def test_all_adapters_importable():
    # it imported, so it must work, right?
    pass

pass is not a test. It's a green checkmark with a body. In a PlannerCritic review I found 57 of 65 assertion files were in the wrong format, so the harness quietly returned 0/0 and called it a pass. The suite was green because it ran nothing.

If a module returns 0/0, treat it as an error, not a pass. Zero assertions is not a quiet success.

2. Letting the Model Grade Itself

This one felt like a model problem. It wasn't.

The model scored precision 1.00 and recall 0.02 and still passed. It had found a degenerate trigger it could fire forever and collect the credit. My first instinct was to tune the prompt. That was the wrong layer, and I kept tuning for a day.

My 3B model learned to match the literal string "step_1" because the reward function paid for overlap, not for correctness. It wasn't learning to review. It was learning what got paid. Full write-up here.

Precision 1.00 with recall near zero is not a safe model. It's a model that found the cheapest way to look safe.

3. Scoring Against the Wrong Denominator

Recall was stuck at 0.087. Same number, two field tests in a row, no matter what I changed. I was about to conclude the model had a ceiling.

I rebuilt the matcher. That was a waste. The matcher was fine.

The real fix was one line of scope: restrict the reference pool to the source domain. The model was answering the right question against a haystack of unrelated gold examples, so it looked bad even when it was mostly right. Once I scoped it, recall went 0.087 to 0.170 / 0.228 with no model change at all. The full numbers are here.

If a metric is stuck at the same low value across every model you try, suspect the denominator before you suspect the model.

4. Optimizing the Layer the Report Named

A week of matcher work moved the metric ten points. Ten points. I'd been fixing the matcher because that's the layer the failing tests kept naming, and I trusted the report more than I should have.

The change that actually moved the needle was six lines in the simulator that generated the failing cases. The score jumped from 20% to 50%. Six lines. I wrote about that week here.

This is the mistake that stings the most, because the report pointed me at the matcher and I believed it.

The layer the failure names is not always the layer that's broken. Prove the origin before you spend the week.

5. Treating a Model-Independent Failure as a Model Ceiling

A local 4B and a cloud model hit the exact same wall. Uniform 20% across both. My knee-jerk was to buy a bigger model. Same wall.

Identical failure rates across wildly different models is almost never a capability ceiling. It's a shared harness bug. I compared a local 4B, a better cloud model, and role separation, and the results were identical in a way that should have been a warning. The comparison is here.

If every model plateaus at the same number, that constant is the thing to fix. Not the model.

6. Mocks That Can't Break

Unit tests green. Real agents failing in the field. I wrote more mocks. Wrong move.

My mock suite reported 9% pass. The zero-mock field test found every single failure the mocks couldn't express. Why? A mock agent always calls the tool it's told to. A real model, sometimes, just answers in prose.

A mock can only fail in the ways you already imagined. Real agents fail in the ways you didn't.

Your mocks are a record of your current imagination, not of the system.

7. Forgetting the Runner

A 1,000-run benchmark died at the 80% mark. I blamed the corpus. I re-ran it. It died again.

The bug was the runner. One hung API call had discarded 60 completed audits, no error record, nothing. I added a per-trajectory timeout, non-retryable timeout classification, a token cap, quarantine on failure, and cancel-on-shutdown. After that, a clean field test ran 4,768 trajectory runs with zero lost sweeps. The field test report is in the repo.

A hang is a data-integrity bug, not a performance bug. A partial sweep still looks like data, and that's the trap.

The One Thing All Seven Share

Look back. Only one of these was actually a model problem, and even that one was fixed in the scorer, not the prompt. The other six were measurement bugs: a suite that didn't assert, a denominator that lied, a layer I misidentified, a classifier shared across models, mocks that couldn't fail, and a runner that dropped data.

I'd rephrase the lesson this way: the reward function is almost always the real bug. You can tune the model forever and the number won't move, because the number was never the model to begin with.

The open question I still can't answer: how do you know your eval is honest? I can name the seven mistakes now. I can't fully trust that I've caught the eighth. A green suite with a hidden measurement bug and a red suite with a real bug look identical from the command line. I don't think there's a clean test for that yet, and I'd rather say that out loud than pretend I closed it.

Which of these did you ship before it taught you? I'll admit to all seven. Tell me yours.

Code and receipts: agent-eval-forge · CauterRule · planner-critic-engine — all MIT, all public.

I Cut 2,490 Agent Test Runs to 206 and Kept the Same Coverage

Debashish Ghosal — Tue, 22 Sep 2026 12:19:00 +0000

The full matrix was 83 agents × 30 scenarios = 2,490 runs. Each one a real LLM call, 30–80 seconds. At 10 workers that's about 2.7 hours, and in practice 4–5× that once you're debugging, so we're talking well over 10,000 calls. Serialize it and it's twelve days.

I ran 206 of those. Not because I was lazy, and not because I was cutting a corner. The other 2,284 were paying real money to re-prove behavior that deterministic tests had already proven, with a local 4B model, which is the most expensive way to run a test that doesn't need a model. Same coverage, one afternoon.

If you've ever stared at a test matrix you didn't want to pay for, keep reading. I work on agent-tooltrust, an open-source gate for AI agent tool calls, and this was the design that let us run the field test at all.

The Trap Was a Whiteboard Diagram

The engine was already proven. 2,490 deterministic assertions, zero LLM calls, every decision path exercised. It was done in the boring, correct way.

But the field test kept pulling me back to the cross-product, because the cross-product is what "thorough" looks like on a whiteboard. 83 × 30. Write it down and it feels safe. Every agent, every scenario, no gaps, nothing to defend in a review. It's also a combinatorial trap, and the bill shows up in two places you don't want: compute cost, and the days it eats while you wait.

The cross-product also wins meetings, and that's the part nobody warns you about. Anyone can propose a clever sampling design, but 83 × 30 is a number you can defend in a room without explaining covering problems. So it keeps getting picked, not because it's right, but because "I tested everything" is an easy sentence. "Every scenario once, every decision type once per framework" is the true sentence, and it took me a week of staring at the matrix before I realized I was optimizing for how the number sounded in review.

The uncomfortable question was: what is a real LLM call actually for here, if the engine is already proven? And once I wrote that down, the answer was smaller than the whiteboard.

The Field Test Had One Job

The field test's real job was adapter proof. Does each framework correctly surface allow, audit, escalate, and deny in a real agent loop? That's it. That's the one thing a deterministic test can't do, because it needs a model that sometimes misbehaves.

That's a covering problem, not a cross-product problem. Once the engine is framework-agnostic, you don't need every agent × every scenario. You need every scenario covered at least once, and every decision type proven at least once per framework. Everything else is re-proving the same cells.

The Design

Plan	Runs	Coverage	Result
Cross-product	2,490	every agent × every scenario	~12 days, 10k+ calls
A — one scenario per agent	83	all 30 scenarios, 10 frameworks, 5 agent classes	83/83 (100%)
B — per-framework decision-type proof	123	all decision types within each framework	116/123 (94%)

Total: 206 runs instead of 2,490, a roughly 12× cut for identical coverage.

Plan A covers breadth. Every scenario is exercised by at least one real agent, across 10 frameworks and 5 agent classes. If a scenario is going to behave differently under a real model, some agent in the 83 is going to hit it. Plan B covers depth. Within each framework, every decision type is hit at least once, so we know the adapter can express all four verdicts, not just the easy ones. Together they prove what the cross-product was trying to prove, without paying for the cells that add no new information.

The full breakdown, including the scenario-to-agent mapping, is in the v0.1.1 field test report.

What Worked, and What Didn't

What worked: the split. Breadth and depth are different questions, and each one has a cheap, correct answer. The moment I stopped asking "how many cells do I need to fill?" and started asking "what is the LLM call for?", the total stopped being scary. 83 and 123 are both numbers I could defend in a review. 2,490 was a number I was too embarrassed to defend, which is how I knew it was the wrong one.

What didn't work: the seven failures in Plan B. And here's the part worth stealing. All seven were not-available. The LLM didn't call the guarded tool. Not one of them was unexpected-decision, the engine returning the wrong verdict. A mock agent always calls the tool it's told to. A real 4B, given five tools at once, sometimes just answers in prose.

That distinction is the whole post. unexpected-decision means the gate is broken. not-available means the model behaved like a model. If you don't separate those two, you'll either ship a flaky gate that fails for reasons you can't fix, or ship a gate that's silently too loose because you decided to "ignore" the model's weird behavior.

It's easier to believe when you've seen the prose. The scenario asked the agent to use a guarded tool. The model wrote, in perfect English, what it intended to do, and never called anything. A deterministic test and a mock agent will never produce that sentence, because neither of them has a language center. The gap between "the gate works" and "the gate works on a model that sometimes decides talking is easier than calling" is exactly the gap only a real agent can show you. That's what Plan B was paying for, and it's the coverage the cross-product would have bought twelve times over.

Where This Breaks

I want to be clear about the assumption, because it's the part that doesn't travel. A covering design assumes independence between the engine and the adapter. That held here because the engine is framework-agnostic. If your system has cross-cutting interactions between agents and frameworks, the cross-product may actually be cheaper than finding the gap later, and I'd run it before you cut the matrix.

And the $0 failures are only trustworthy when code review already caught the real bugs. The field test is the second line of defense, not the first. Pretending it's the first is how a green coverage number becomes a false one.

The Open Question

I still can't fully answer how to decide what only a real agent can prove, versus what deterministic tests can. I made a judgment call here, and it happened to be the cheap one. I don't know a principled way to know that in general, and I'd rather own that gap than oversell the 206.

How do you decide where your deterministic tests stop and your real-agent tests begin? I'd genuinely like to hear how other people draw that line.

Code and receipts: agent-tooltrust · field test report · field test plan · design decisions — all MIT, all public.

1,558 Tests Green and No Auth: The Tests That Never Actually Ran

Debashish Ghosal — Sat, 19 Sep 2026 20:43:50 +0000

A test named test_all_adapters_importable asserted nothing. It would pass forever, even if every adapter was broken.

57 of 65 assertion files were in the wrong format, and the harness returned 0 / 0 without a whisper.

If a green suite makes you relax, this post is going to un-relax you.

Three Times a Green Suite Hid a Real Failure

1. The test gutted to pass. In planner-critic-engine, a test literally had a pass body:

def test_all_adapters_importable():
    # if it imported, it's fine — except this proves nothing
    pass

It was caught in code review, not by CI, and only before the LLM sweep because a human read it. Issue #236.

2. The harness that returned 0 / 0 and called it green. In the same repo, 57 of 65 assertion files were in the wrong format. The harness parsed them, found zero assertions to run, and returned 0 / 0 — which it treated as success. The suite was green because it had executed nothing.

3. The auth guard that never ran. In CauterRule v0.3.0 the suite reported 1,558 tests green. The MCP HTTP bearer-auth guard never ran, because an import was swallowed. Every unit test passed. The only reason we found the unauthenticated store read was a Docker field test running the real thing in a real container.

The Dangerous Thing Wasn't the Bug

In all three cases the bug mattered, but it wasn't the scariest part. The scariest part was the confidence.

We treat "tests pass" as evidence. Sometimes it's evidence that the harness silently skipped a module, or that an import was swallowed, or that a fixture file was parsed into an empty set. The suite stays green and the system ships with an unauthenticated read path.

The Fix Is a Meta-Test, Not More Tests

The fix is not to write more tests. It's to assert that your tests actually ran and actually asserted.

def test_suite_is_not_empty():
    results = run_assertion_files("tests/assertions/")
    assert results.executed > 0, "harness ran zero assertions"
    assert results.assertions > 0, "assertions parsed to empty set"

Two rules fall out of this:

CI must fail when a module produces zero results. 0 / 0 is an error state, not a pass.
Distinguish "did not run" from "ran and passed." A swallowed import and a skipped module should be loud failures, not silence.

That single distinction is the difference between a flaky gate and a strict one.

The Honest Limitation

Meta-tests add process, and process can rot — a meta-test that stops checking is just another green checkmark. And no amount of test discipline catches the false negatives you never thought to test for. This reduces the class of "green but broken." It does not eliminate it.

But it does close the worst category: the test that never ran and told you everything was fine.

What's the last green build you caught lying to you? I now trust a green suite about as far as I can read its raw output.

Repos and receipts: CauterRule · CauterRule v0.3.0 field test report · PlannerCritic failure-mode register · agent-tooltrust design decisions — all MIT, all public.