DEV Community: Abhinav Pangaria

Making Verification Visible

Abhinav Pangaria — Tue, 22 Sep 2026 03:34:10 +0000

The hackathon gate had one output: a line in the terminal saying a task passed, failed, or needed a human. That line is honest, but it is small. It does not tell you what the agent touched, which requirements have a test behind them, which of those tests have ever passed in a browser, or why a task has been blocked since last night.

The rebuild's answer was a dashboard, and it is the part people miss when they hear GuardianKane described as a Stop hook. This is a tour of it, using screenshots from the ecommerce demo, a real project built through the loop and committed with its unedited history. Part 1 covers how the system got here.

Four tabs

The dashboard runs at localhost:4173 and has four tabs: Code graph, Memory graph, PRD graph, and Kane activity. An "Ask GuardianKane" chat panel sits on the right of every tab. The graphs are drawn with cytoscape; the panels poll small JSON files that the hooks and the graph build write into .testmuai/.

Code graph

The v2 design spec defines three kinds of node and four kinds of edge:

code nodes are files, from an import scan with madge
feature nodes are tasks from the tracker
claim nodes are Kane's use cases
imports and calls connect code to code
touches connects a task to every file the agent edited while working on it
about connects a claim to the files it mentions

The legend on this screenshot counts 42 code nodes, 7 features, 6 claims, 42 import edges, 41 touches, and 13 about edges. The touches edges are the dormant data from Part 1: the PostToolUse hook had been recording them since the hackathon version and nothing read them until this graph did. They are also the reason the graph is useful. The orange cluster on the right is what the agent actually changed per task, drawn next to what the PRD says those tasks are about.

Nodes carry a phase (P0 to P5 here) and a test-quality ring: strong, weak, or untested. Clicking one or more rings isolates them.

Focus and selection

This is what "pull-based" from Part 1 looks like in practice. You select nodes; the sidebar tells you how they connect (here, nine nodes joined by eight imports, flagged as an integration test candidate); and "Generate tests" asks Kane for tests scoped to that selection instead of to the whole PRD. "Ask about tests" goes to Kane's recorded explanations. The same sidebar holds the review card, the trace for a selected task, an on-request whole-app bug sweep, and quick-generate.

One thing this screenshot shows that I did not intend: the selection includes agent-bridge.js, graph-build.js, kane-context.js, and tracker.js. Those are GuardianKane's own files. The installer copies them into the target project, and the graph scans the project, so the tool shows up in the graph of the app it is checking. The graph build takes an --exclude flag, but nothing excludes the installed files by default. That default is the fix.

PRD graph

The PRD graph is Kane's claim graph for the project: every acceptance criterion as a node, grouped under the feature it belongs to, and every gap as a red marker. With the Gaps and drift panel open, the same tab shows the number I think matters most in the whole dashboard:

Design 100% (21/21 ACs). Proven 53% (10/21 ACs). Every acceptance criterion has a test designed for it. Just over half have a test that has actually passed in a browser. Those two numbers are the difference this whole project is about. The gate is there to make the second one move, and the panel keeps the gap visible: 14 pending gaps, each with the Kane command that would close it. This panel exists because of Phase 5, which found that kane-cli cover gaps --json already computed all of it and nothing called it.

A project that reports only the first number is reporting what it intends, not what it has.

Memory graph

The memory graph renders knowledge memory: run history keyed by the set of files a verification touched, pass or fail, with timeouts and infra errors left out. This is the "memory of where" that Part 1 says the hackathon version lacked.

It needs reading with care. styles.css shows 15 runs, 12 fails and 3 passes, and "currently fixed". Those are not 12 failures of a stylesheet. A stylesheet is touched by nearly every verification, so its history is a record of the tasks that went through it. And the one annotated failure is automation_bug at 0.96 confidence: the test expected the wrong email, not the app showing it. The family is stored, so the information to tell app bugs from test bugs is there. Nothing filters on it yet.

Kane activity

The Kane activity tab is kane-activity.log rendered newest first, with one colour per outcome.

Its first version had the colours wrong: the classifier checked for "passed" before "failed", so a line like "FAILED ... 2/3 passed", which appears below, rendered as a pass. That was fixed in 86b3d09, from a dogfooding pass.

The chat is wired into the live session

"Ask GuardianKane" is not a separate chatbot. A message typed there lands in the Claude Code session that is building the app, as a user turn. The code that does it is about fifteen lines in dashboard/lib/agent-bridge.js: connect to the Unix socket named in CLAUDE_CODE_MESSAGING_SOCKET, write an auth line with the session's token, write the message. The comment above it is blunt about where that came from: the protocol was "reverse-engineered from strings embedded in the claude CLI binary itself; it is not documented anywhere public". The dashboard has to be launched as a child of the session to inherit the socket and token.

That makes it fragile in a way I want to be upfront about. It depends on an undocumented interface that could change in any Claude Code release, and I have no guarantee it will keep working.

The first version also had a problem that was obvious once anyone used it: messages went in and nothing came back. The chat relied on the agent remembering to post a reply to the dashboard, and it did not. Commit b2356c4 moved the reply out of the agent's hands. Sending a message now writes a chat-pending.json marker; the Stop hook, which runs at the end of every turn anyway, checks for the marker and posts back the verdict text it already computed. In the code graph screenshot above, the last chat bubble ("T-T5 failed verification (attempt 1/3) ...") is in the Stop hook's own denial format, which is the text the auto-reply posts. The agent did not write it. The gate did.

It is the same move as the rest of the project. Do not rely on the agent to report on itself; have the thing that checks it do the reporting.

What the demo's log says about the gate

The dashboard earned its keep by showing me bugs in GuardianKane, not just in the app. The ecommerce demo's committed kane-activity.log is 103 lines, and some of its lines are wrong about themselves.

"FAILED ... reason: all members passed". T1's first batch, on August 29:

[T1] scripted test FAILED (attempt 1/3). summary: testrun batch: 2/3 passed. reason: all members passed

T2 has the same shape at 3/4, and T5 at 0/3. Commit a6a6dc1 (August 30) traced two causes in the runner. A batch killed mid-flight fell through to a reason that claimed every member passed. And kane-cli's process exit code can be non-zero on a full pass: the commit records a live run that "exited 1 with status: 'passed'", which had produced a false KANE_FAILED. The fix trusts Kane's structured verdict over the exit code. The commit is timestamped 12:16 IST; T5's contradictory line is at 14:17 IST the same day. The installer copies the hooks into the project, and I do not have data on which version the demo was running at that moment.

"AC EVIDENCE CHECK FAILED" after a 3/3 pass. Later on the 29th, T1's batch passed 3/3 and was still failed by the per-AC evidence check, three times in a row, with "no evidence pack path reported" and then "no matching evidence pack found". Commit 9181821 found the gate matching evidence on the wrong field. Kane's result.yaml has a definition_id, the content hash in the test's frontmatter. The check looked for an assurance_id, which the batch output did not contain, so it "silently failed the evidence check on every batched task". The commit message names T1's failures as the case it matches. For those three strikes, the app was fine and the checker was wrong.

"all tasks KANE_VERIFIED. Build complete." At 19:01 UTC on the 29th, after T2 was verified, the log says the build is complete. The next line starts T3's tests, and T4 and T5 follow the next day. I do not have data on what the tracker held at that moment; the log alone shows the gate declaring done while work was still being verified.

"FAILED ... reason: Objective completed". After T3's tests were rewritten, its first run on the new files is logged as a failure whose summary describes a successful sign-up and whose reason is "Objective completed". I do not have data on why that run was classified as a failure; it is not one of the cases the commits above describe.

The rest of the log is the loop working as designed. T3 reaches three failures, the last with assertion_failed: @ step 3, escalates to a human, and after its tests were rewritten goes on to pass and verify. T4 goes 0/3, 1/3, 3/3, then verified. T5 runs into three kane-cli timeouts (exit null, state unchanged, no strike charged), then three counted failures: the first is the "all members passed" contradiction above, and the last two are on the reject-invalid-card-details test. It ends the committed log at BLOCKED_NEEDS_HUMAN. The chat in the code graph screenshot, taken on August 31, shows T5 picked up again after that.

So the closed loop in the demo is real, and it is not tidy. Two of the escalations in its history were at least partly the gate's own bugs. That is the lesson of who verifies the verifier again, and the reason the dashboard renders the raw log instead of a summary of it.

The dashboard had its own bugs too

One more, because it is the kind a verification tool should be embarrassed by. Task titles, verdict summaries, AC references, and log lines were written into the page with innerHTML unescaped in three renderers: the review card, the trace panel, and the stuck-tasks panel. Commit 1c9f476 calls it what it was, a stored XSS through server-controlled fields, and escapes them. Some of those fields come from Kane's output and some come from the agent, so "server-controlled" here means "written by software I do not fully control".

The same dogfooding pass (86b3d09) fixed a "show N more" toggle on the drift list that never worked because of an inverted boolean, and started reaping orphaned Chrome processes that kane-cli's runs left behind.

What visible buys you

A pass or fail line tells you whether to keep going. The dashboard answers the questions that come after it: what did the agent touch, which requirements are proven and which are only designed, which failures were the app and which were the checker, and what has been stuck since yesterday. None of that makes the agent more honest on its own. It makes it cheaper for a person to check, which in my experience is what decides whether anyone checks at all.

The full current build is in the demo video, the code is at github.com/18Abhinav07/adventures-with-kane, and the ecommerce demo with its history is at github.com/18Abhinav07/guardiankane-ecommerce-demo. GuardianKane is built on Kane CLI by TestMuAI.

Part 2 of 2. Previously: the second build was mostly wiring. The hackathon version, and the experiments behind it, are in Your agent might actually lie to you.

Part 2 of GuardianKane: It Will Not Let Your Agent Lie. The original, with the full series navigation, is on my site.

The Second Build Was Mostly Wiring

Abhinav Pangaria — Tue, 22 Sep 2026 03:34:09 +0000

My first series, Your agent might actually lie to you, is about one shape of GuardianKane: a Claude Code Stop hook that will not let the agent finish until Kane CLI has checked the work in a real browser. That shape placed 2nd in the TestMuAI Kane CLI hackathon, and four paired experiments stand behind it.

It is not what GuardianKane is today. The hook is still there, but it is one piece of a larger system: a reviewed PRD, a phase model, a live scope guard, a file lock, a secret scan, and a dashboard with graph views and a chat panel wired into the running Claude Code session. This is the story of how it got there. Part 2 is the tour of what it looks like.

Three shapes, one repo

The repo's JOURNEY.md records each shape against the commits that built it.

Era 1 was a planning tool. The first four commits are a design spec, an implementation plan, a task-tracker library, and a Claude Code skill with a thin kane-cli wrapper. You gave it a PRD and got back a task-tracker.md: tasks with ids, titles, PRD references, and dependencies. Nothing checked whether a task marked done was done.

Era 2 was the gate. The same afternoon, 896bc57 added the Stop and PostToolUse hooks with the full decision table, and a day of running the loop against real apps turned it into the product the first series describes: scripted replay, a defect sweep, bug memory, an installer, and the four paired experiments. Two demo videos come from this era, a 14-minute walkthrough and a 3-minute cut.

Era 3 started five days later. It opens with a new dashboard on August 26 and then runs through twelve numbered phases, each with its own design spec and implementation plan under docs/superpowers/. The current build has its own demo video.

Why rebuild something that worked

The v2 design spec opens with the problem, and it is not "the gate was broken". It is that asking a coding agent to generate tests for a system produces "on the order of hundreds of assertions, all green", that do not exercise real browser behaviour and do not map to what the PRD asked for. The spec calls these AI-slop tests and names two separate things wrong with them.

The first is that they are push-based. The agent reads the PRD once, generates everything it can think of, and stops. v2 makes verification pull-based: a person (or later the Stop hook) selects concrete nodes in a graph of the codebase, and only those nodes' claims get tested.

The spec is careful about how much that buys, and I want to keep that care here. An earlier draft said v2 fixed both problems "by construction". An independent review caught it, and the spec now says pull-based targeting narrows scope, not depth. A vague instruction fed to the same browser agent can still produce a shallow pass. What v2 guarantees is that verification is aimed at something a person deliberately pointed at, not that any given run is rigorous.

The second problem is no memory of where. Era 2's bug memory could say "this looks like a bug we have seen before" by comparing failure text, but it never recorded which files were involved, so it could not say "this exact set of files was fixed on the 20th and has regressed". v2 keys its knowledge memory by the set of graph nodes under test.

The phase that fixed every fresh install

Before any new feature, the plan for Era 3 audited what Kane CLI already did and what GuardianKane was duplicating. Its conclusion became the architecture:

GuardianKane's job narrowed to orchestration. Kane owns the claim graph, coverage, reconcile, and explanations; GuardianKane owns tasks, phases, the gate, and the dashboard.

The same audit found the bug that became Phase 0. Kane CLI will not generate tests for a use case that has not been reviewed. It refuses with uc-1 is unreviewed, and the fix is kane-cli context review --approve. The plan records that context review was "never called anywhere in the current codebase". So on any fresh install, test design failed on the first use case. I do not have data on why my Era 2 runs never tripped over it.

Phase 0 (39aaeaa) wires the review step in, so every grilled use case and every generated test is approved before it counts. That is also the honest version of what "PRD grilling" means in the current build: a requirement is extracted, questioned, and reviewed before anything is built against it.

Twelve phases, and what most of them had in common

Phase	What it added	Built on
0	Review gate	a Kane command nothing called
1	Per-AC evidence and tamper detection	Kane's sealed evidence packs
2	Phase model in the tracker	the existing tracker schema
3	Context injection before a task starts	`context explain` / `design explain`, which make no model calls
4	Live scope guard	the PostToolUse hook's file-touch record
5	Gaps panel	`kane-cli cover gaps --json`, which nothing called
6	Review card with per-AC acknowledgment	the task's diff and evidence
7	Reconcile-powered sync	`kane-cli maintain reconcile --plan`
8	Trace panel	the activity log, never rendered
9	Stuck-tasks panel	attempt history already written per task
10	File-level task lock	the same file-touch record
11	Secret scan and negative-path nudge	the task's scoped diff
12	Browser review, PRD graph, quick-generate, evidence viewer, Playwright export	the rest of Kane's surface

Read down the last column. The phase specs say it themselves, over and over: "already computes", "already fires on every Edit/Write", "nothing in the codebase calls it today". Phase 5's spec confirms the gaps command was unused with a grep that returns zero results. Phase 7 replaced a PRD diff I had written by hand with Kane's own reconcile, because mine was "reimplementing something Kane already computes".

The v2 spec has a phrase for the pattern: the first place v2 "activates dormant data rather than adding new collection". It is about the touches edge in the code graph. Since Era 2 the PostToolUse hook had recorded every file the agent edited, per task, and nothing downstream had ever read it. In Era 3 that one record feeds the code graph, the scope guard, the trace panel, and the file lock.

That is the lesson I would carry to the next tool I build on top of an agent: before adding collection, grep for what is already collected and never read.

Where the phases were honest about their limits

A few specs say plainly what their phase does not do, and those are the lines I trust most.

Phase 4 writes scope drift to its own scope-status.json instead of the existing graph-status.json, because the graph rebuild would overwrite the flag "often before the dashboard's 5s poll interval catches it". The spec calls it "a real write race, not a style preference".
Phase 8 found that the PostToolUse hook exits early for anything but Edit and Write, so "nothing is recorded for Read, Grep, Glob, or Bash". Half the trace it was asked to draw did not exist yet.
Phase 10 only warns. There is no PreToolUse hook in the codebase, and PostToolUse fires after the edit "has already happened, so it cannot block the edit itself". The spec implements the warn half of warn-or-block and says so.

Who built the second build

This part would not be honest without it. The specs for Phases 3 to 12 carry a status line saying they were approved "under the standing full-autonomy delegation", self-reviewed or given a separate architect-style review pass, with no per-phase sign-off from me. Of the 50 most recent commits in my local clone, 29 carry a Claude Sonnet 5 co-author trailer.

So the rebuild of a tool that exists because agents claim done too early was itself largely built by an agent working under a delegation. What kept it honest was the same thing GuardianKane asks of any agent: a written spec per phase, a plan, and a test suite that has to stay green. The phase specs record 241 of 241 tests passing after Phase 1 and 387 of 387 after Phase 11. I ran the suite again for this post: 36 test files, 460 of 460 passing.

That count is unit tests on GuardianKane's own code. It is not evidence that any app built through the loop is correct. That evidence is the browser runs, and Part 2 shows some of them failing for reasons that had nothing to do with the app.

What came after the twelfth phase

The last days of August were not new features. They were fixes found by running the whole thing against a real project, the ecommerce demo: the evidence check keyed off the wrong field, a runner that trusted kane-cli's exit code over its own structured verdict, a dashboard chat that never replied, and a stored XSS in the dashboard's own rendering. Every one of those is in Part 2, with the log lines that exposed them.

Part 1 of 2. The hackathon version is the first series, starting with the broken chart. Next: making verification visible, the dashboard, the graphs, and the chat bridge. The code is at github.com/18Abhinav07/adventures-with-kane.

Part 1 of GuardianKane: It Will Not Let Your Agent Lie. The original, with the full series navigation, is on my site.

Who Verifies the Verifier?

Abhinav Pangaria — Tue, 22 Sep 2026 03:34:07 +0000

The premise of this series is that you should not take an agent's word that it is done. So I built a gate that makes Claude Code prove each task in a real browser before it may stop. The browser is driven by Kane CLI, which is itself an agent.

The obvious question arrives late and uncomfortably: why would I take that agent's word either?

This is the part of the story where the gate turns around and looks at itself. It has three layers: the verifier's failures, my own, and the rule that decides when to stop asking.

Twelve tasks, eight failures, one bug

ORBITAL is the densest build in the project, the one with the doubled chart from Part 1. Every task went through the gate, and the Stop hook wrote each attempt to an append-only activity log. Here is the whole build, from that log.

12 of 12 verified. One sweep catch. Six overrides.

Eight tasks failed a check at least once. T1 failed on an assertion timing mismatch after the navigation it was checking had already happened, and passed on retry. Six more (T2, T3, T4, T9, T10, T11) failed badly enough to need a human-reviewed override. For every one of the six, kane-cli's own triage came back confirmed: false, family automation_bug. The causes were specific:

T2, the performance chart. Three attempts, three different failure points. One opened the wrong page entirely. Triage pointed at flaky headless hover targeting and an ambiguous unlock-state assertion.
T3, the allocation donut. The test asserted a hover effect without ever hovering the segment first. The script was missing a step.
T4, the holdings table. A legitimate layout fix invalidated recorded replay baselines. Re-recorded with --author, all eight test files passed.
T9, T10, T11. The feature steps passed, then a redundant final re-check stalled. On T11 the browser agent oscillated between two buttons until it detected a cycle in its own plan.

The eighth was T7, the sweep catch from Part 1. That was the one real problem in the app across twelve tasks.

Eight failures, one bug. Read by exit code, all eight look the same.

The rule I ended up with

Every raw failed result gets read against the verifier's structured verdict: verdict.confirmed, family, category, and confidence. Never the exit code. Then, before an override is allowed:

Re-run the failing test alone, not inside a batch.
Get kane-cli's triage on the repeated failure.
Find direct evidence the feature works: a step in the same run that passed, a fresh recording, or a manual check in a real browser.
Only then clear it, with the rationale written into the log next to the evidence.

The same held across the other experiments. On the booking studio, six toast-timing failures were Kane's agent being too slow for a four second toast. On the todo build, a persistence check looked under the storage key the other build happened to use.

A slow verifier, a brittle assertion, a stale recording. None of those is the app. All of them look exactly like the app from outside.

Then I re-read my own write-up

This is the part I did not expect to write.

While putting this series together I went back to the repo's build log, OBSERVATIONS-AND-REPORTINGS.md, and checked each ORBITAL claim against the raw activity log. My write-up had done to the verifier's failures exactly what the verifier had done to the app.

What my write-up said against what the log said.

The notes say the sweep "caught a real state-tracking defect" on the chart at T2. The log shows no sweep finding on T2 at all, only three scripted failures that triage called automation. The notes credit the sweep on T3 and T4; both were test problems. The notes call T9 "a genuine race"; the log shows all four export toasts confirmed in the same run and a redundant step failing afterwards.

Kane's bug_title fields made this easy to get wrong. Triage titles read like bug reports: "Replay misclassifies unlocked chart state". Lift that string out of its context, forget the confirmed: false next to it, and it reads as a finding. I lifted them. The first draft of Part 1 repeated them, before I checked.

The repo's own evidence gif has the same problem. The README captions this one as a caught defect:

A failed run on the chart lock, evidence pack 31fcd34b. Every failure on the chart task was triaged automation_bug, so I now read this as the verifier failing, not the chart.

The overrides I am least comfortable with

Look at who cleared each override. T2 was recommended by a second Claude session and approved by me explicitly. T3 needed triage evidence and then my approval. T4, T9, T10 and T11 were cleared "under standing self-override authorization": a standing grant, not a fresh decision each time.

The evidence supports each of those calls. But the process got weaker as the build went on, and the log shows it. A human in the loop who has pre-approved the loop is not really in it.

Three strikes, then a human

All of which forces a design question I had to answer in code before I could answer it in prose. When the gate says no, how many times do you send the agent back? An agent told "try again" will try forever. It does not get tired or embarrassed. Somebody has to decide when the loop has stopped producing information and started producing noise.

GuardianKane's answer is three.

Every gate that can fail a task follows the same shape: increment attempts; under three, set KANE_FAILED and deny the stop with a reason; on the third, set BLOCKED_NEEDS_HUMAN. The details around the number matter more than the number:

A failure always routes back to IN_PROGRESS. There is no edge from a failed check back to CLAIMED_DONE. The agent cannot re-assert done on the same code.
Escalation is an allow, not a deny. A deny with no retries left is an instruction to keep going with nothing new to say. The allow ends the session and leaves the task blocked where a person will see it.
Tampering skips the queue. If a test lost an assertion for a requirement that did not change, that is a test being weakened to pass, and it escalates on the first occurrence. Retries are for honest mistakes.
The verifier's failures are free. When kane-cli exits with an infra error or a timeout, state is left alone and no strike is charged. Given everything above, a counter that billed the agent for Kane's bad hour would escalate working code.

The counter that went backwards

The clearest log of the cap in action comes from a later project, an ecommerce demo built with the rebuilt GuardianKane and committed with its unedited history. The three-strike rule is unchanged there. Here is its first task, T0:

Time	Event
20:15:30	sweep FOUND ISSUE (attempt 1/3): (no summary)
20:27:58	SECRET SCAN FAILED (attempt 1/3): server.js, high-entropy value assigned to "secret"
20:29:56	sweep FOUND ISSUE (attempt 2/3): (no summary)
20:30:23	sweep FOUND ISSUE (attempt 3/3): (no summary)
20:30:23	BLOCKED_NEEDS_HUMAN after 3 sweep failures
22:11:02	sweep found no issues, KANE_VERIFIED

The escalation works. And the log taught me three things.

The counter went backwards. The first sweep failure is attempt 1. Twelve minutes later the secret scan failure is also attempt 1. The hook only ever increments attempts; nothing in it sets the field back to zero. But the counter lives in the task tracker, a YAML block in a markdown file the agent also edits, because the agent is the one who flips tasks to IN_PROGRESS. Nothing guards that field. I do not have data on what reset it in this run. The log only shows that it was reset.

That is the most important finding in this series. "Only the hook writes verification states" is a rule the agent is told, not one the file system enforces, and the same is true of the retry budget. A cap the capped party can edit is a suggestion. The fix is to keep attempts and the verification states in a file only the hook writes, and treat the tracker as a view.

The last two sweeps were not a browser looking at a page. Attempt 3 started at 20:30:21 and reported an issue at 20:30:23. Two seconds, no summary. A sweep drives a real browser against a running app; a two-second verdict with nothing behind it is not evidence about the app. The escalation was reached mostly on verdicts like that, which is the lesson from the top of this post again.

The human step is invisible. T0 sits blocked for about an hour and a half, then comes back clean. Whatever happened in between, a fix, a restart, a tracker edit, is not in the log, because the log only records what the hook does. For an audit trail, the most consequential step, a person deciding to reopen a blocked task, is the one it cannot see.

Why three

I do not have data that three is the right number. I did not run the loop with two or five and compare. What I am more confident about is the structure around it: honest failures retry, tampering does not, the verifier's own failures do not count, and the budget has to live somewhere the agent cannot write. The hackathon version got the first three right.

What this changes about the thesis

It does not weaken it. It sharpens it.

The original claim was "do not trust the agent's self-report". The better claim is "do not trust any unverified claim, including the verifier's, including your own summary of the verifier". Each layer needs its structured evidence kept next to its conclusion, so the next reader can check the step instead of the sentence.

In practice: read structured verdicts, not exit codes. Keep confidence and family next to every finding, everywhere it is quoted. And go back to the raw log when you write it up, because the summary is where the drift happens.

The raw log was right the whole time. Everything that went wrong went wrong in a summary.

That is also where the hackathon version runs out. A pass or fail line in a terminal cannot show you what the agent touched, which requirements are only designed and which are proven, or which failures were the checker. Answering that took a second build, and it is a different story.

Part 3 of 3, and the last. Previously: the requirement nobody wrote. The story continues in GuardianKane: it will not let your agent lie, starting with the second build was mostly wiring.

Part 3 of Your Agent Might Actually Lie to You. The original, with the full series navigation, is on my site.

The Requirement Nobody Wrote

Abhinav Pangaria — Tue, 22 Sep 2026 03:34:05 +0000

The smallest PRD in my experiments has a section called "Priority badge". It asks for a badge on each task card, keeps the existing layout, and then says this, which is the whole specification of priority:

just with a small badge showing High/
Medium/Low priority somewhere on the card, defaulting to Medium.

Read it again and try to answer one question: how does a task become High?

It does not say. Three values are allowed, one is the default, and there is no mechanism for reaching the other two. Nobody wrote that part down, and I did not notice until a tool asked.

Two builds, one gap

This was the first of four paired experiments. Same PRD, same starting commit, same operator. todo-kane went through the full GuardianKane loop. todo-baseline was a fresh Claude Code agent given the PRD verbatim and told "Build this. Let me know when it's done." No hooks, no gate, no follow-up.

Both builds worked. The baseline passed the add, complete, delete, and default-badge tests from the gated build's own suite. On a CRUD app this small you would expect nothing else. Then I pointed the explicit-priority test at it, and Kane's verdict came back:

confirmed: true
severity: major
"The high-priority task is displayed with a MEDIUM badge instead of High."

There was no way, anywhere in the baseline, to set a task to High or Low. You do not find that by reading the code, because the code is fine for what it does. You do not find it by clicking around either, because every task shows "Medium" and Medium is a valid value. The app looks correct right up until someone needs a High.

Where the question came from

The gated build did not get this right because it was smarter. It got it right because, before any code existed, GuardianKane's start flow ran a grilling pass over the PRD: a structured conversation whose only job is to find every place the document is ambiguous and force a decision while deciding is cheap.

One question during grilling became a convention, a task, and a test.

The question was the obvious one, once someone asks it. The answer was deliberately small: a !high or !low suffix on the title sets the priority and is stripped from the displayed title. That decision became task T5, and Kane generated a real browser test around it. The step that gates the task, verbatim from the repo:

Assert that the task card titled {{high_priority_task_title}} shows a visible
priority badge whose text is exactly High, and that the same card still shows its
completion checkbox and delete "x" control.

T5 could not reach KANE_VERIFIED until that held in a real browser, so the gated build implemented the suffix. The unassisted build never had the question asked, and resolved the ambiguity the way ambiguity always resolves under pressure: in the direction of least effort.

Why an agent will not ask

I do not think this is specific to Claude. It is what one-shot building does to any gap in a spec. The instruction was "build this", and the spec was satisfiable as written: every card has a badge, the badge can show Medium, Medium is the default. An agent optimizing to finish will not invent a requirement that makes the task bigger, and it will not stop to ask, because stopping to ask looks like not finishing.

A human engineer would probably have asked in standup. The grilling step moves that standup question to before the first line of code, and then turns the answer into something executable. A decision in a chat log can be forgotten. A decision in a test file cannot be skipped.

One smaller finding from the same experiment keeps me honest about what that buys. Re-running the priority test against the gated build itself, Kane flagged that the badge's CSS uppercases the text, so the screen reads "HIGH" while the DOM says "High". It was confirmed: false, severity minor, and it had not failed the gate originally. A verified task is verified against what was asserted at the time, not against every assertion anyone could write later. Verification is only as wide as the questions you asked before it ran.

So I built a trap

The fair criticism of the todo result was that it was too forgiving. The gap only existed because one PRD happened to leave an ambiguity. I wanted a task that would stress the gate itself.

A room booking widget: pick a start and end time, create a booking, reject any booking that overlaps an existing one. The PRD is explicit about the edge:

If the new range overlaps any existing booking, reject it with a visible
error and do not create it. Two bookings that merely touch (one's end
time equals the other's start time) are NOT overlapping and must be
allowed.

This is the textbook off-by-one. Write <= where you meant < and the widget rejects perfectly valid back-to-back bookings. It is invisible unless a test builds two bookings that share a boundary, and it is exactly what I expected a one-shot build to get wrong.

Both builds got it right. booking-kane used aStart < bEnd && aEnd > bStart. booking-baseline used aStart < bEnd && bStart < aEnd. Same logic, operands in a different order, both strict. The gated build ended with a verdict-confirmed record across all 12 scored tests. The baseline completed 6 of 12, including the trap, and passed every one it completed. No confirmed divergence.

My trap caught nothing.

Why I did not bury it

It would have been easy to leave that experiment out. It produced two things I value more than a dramatic screenshot.

It turned a claim into a fact. Before the cross-test, "the baseline handles touching bookings" was the baseline agent's own self-report: the exact kind of sentence this project exists to distrust. After it, it was a verified result. The code did not change. What changed was whether anyone had to take an agent's word for it.

That is the less glamorous half of what a gate does. People picture it catching bugs. Most of the time it converts "I believe this works" into "a browser watched it work", and when the answer is yes, nothing visibly happens. A good verification system spends most of its life producing null results.

It kept the numbers honest. Six of the baseline's tests never completed, because kane-cli was flaky on this task throughout; one run hung for more than 17 minutes before I killed it. Two of the gated build's raw failures were confirmed: false, automation_bug: Kane's own analysis said the UI had already rejected the bad booking. Scored by exit code, this experiment would have reported bugs in the gated build that did not exist. That thread gets pulled all the way in Part 3.

Then I made the PRD denser

The todo and booking PRDs were thin. Neither gave a one-shot build much room to drop anything. The third experiment kept the booking domain and loaded the PRD with detail: exact hex colors, an 8px spacing scale, corner radii, a header row with 16px padding, a 250ms shake on conflict, an auto-dismissing toast with a progress bar, a 5 second Undo that restores the exact cancelled booking, a persisted theme, and rules for what Escape closes first. Every one of those is checkable, and every one is easy to drop.

The gated build reached KANE_VERIFIED on 8 of 10 requirement groups. The other two, toast stacking and cancel with undo, failed six attempts between them, all six confirmed: false: Kane's browser agent was too slow to act inside the few seconds the toast was on screen.

A manual side-by-side found one concrete divergence, on the unassisted build. Its header rendered with the title and theme toggle flush against the viewport edge. The PRD asks for 16px of padding. The gated build had it. One missed value: real, specific, and modest.

The shape across four experiments

Line the four up by how dense the PRD was, and a pattern shows up that I did not design for.

The gap grew with the density of the PRD.

Sparse PRDs produced small gaps or none. The denser the spec, the more places an unsupervised single pass had to quietly drop something. By ORBITAL, Part 1's build, the unassisted agent had a doubled chart, a wrong alerts layout, and uneven spacing.

Four experiments, one operator, one pair of builds each. That is a pattern worth writing down, not a law. But it matches what I believe from building with agents: Claude is already strong at small, well-scoped builds. A gate earns its keep in the long tail of a dense spec, where "looks right at a glance" and "is actually right" start to come apart. And the most valuable thing it did in these four experiments was not the browser check. It was the question asked before the build.

If you write PRDs for agents

Read every enumerated value and ask how each one is reached. Read every default and ask what the non-default path is. Read every "can" and ask what "cannot" looks like. Those three habits would have caught the priority gap.

Better, do not rely on your own reading. The gap sat in plain sight in two lines of text, and nobody saw it until something was built to look. Put a step in the loop whose only job is to ask, and make its answers executable.

And design experiments that can fail. If every experiment you run says your tool is useful, you have probably only run the ones that could.

Part 2 of 3. Previously: the broken chart. Next: who verifies the verifier?, on the failures that were never the app's fault, and a retry cap the agent could edit.

Part 2 of Your Agent Might Actually Lie to You. The original, with the full series navigation, is on my site.

My Agent Shipped a Broken Chart and Told Me It Was Done

Abhinav Pangaria — Tue, 22 Sep 2026 03:34:04 +0000

The chart looked fine in the code.

The data pipeline was correct. The chart library call had the right shape. Every prop was passed, every array was populated, and nothing about the component would have looked suspicious in thirty seconds of review. If you had put that diff in front of me cold, I would have approved it.

Then I opened the page. The performance chart was rendering the same series twice: two overlapping traces at slightly different weights, a smeared, doubled line that no one would ship on purpose.

The same series, drawn twice. This build reported the task complete.

The agent that wrote it had already marked the task done.

This series is about that word, "done", and what it took to stop trusting it. It starts with a tool I built for the TestMuAI Kane CLI hackathon, where it placed 2nd.

A planner that never checked

GuardianKane did not start as a verifier. The first four commits in the repo are a planning tool: a design spec, an implementation plan, a task-tracker library, and a Claude Code skill with a thin kane-cli wrapper. You gave it a PRD and got back a task-tracker.md: tasks with ids, titles, the PRD lines each one implements, and dependencies.

It was tidy, and it had one hole in the middle. The agent worked through the tracker and moved each task to done when it decided the task was done. Nothing checked. The tracker was a list of claims, and the planner's whole output depended on believing them.

The same afternoon, commit 896bc57 added two Claude Code hooks, and the planner became a gate.

The gate, in one picture

Claude Code has a hook called Stop. It fires every time the agent finishes a turn and is about to hand control back. If the hook prints {"decision": "block"} with a reason, the agent does not stop. It reads the reason and keeps working.

That is the entire lever. Everything else is what I put behind it.

The agent never calls Kane itself. The hook does, every time it tries to stop.

The tracker grew a state machine. The agent may move a task to IN_PROGRESS and CLAIMED_DONE. Only the hook is supposed to write KANE_VERIFIED. The asymmetry is the point: the agent can say it is done; it cannot say it is verified.

When a task is CLAIMED_DONE and the agent tries to stop, the hook drives Kane CLI, TestMuAI's browser agent, against the running dev server in a real headless Chrome. Two checks run per task:

A scripted replay. The test Kane generated for the task, run exactly as authored: kane-cli testmd run <file> --agent --headless --variables <json>, with start_url pointed at the local server.
A defect sweep. No test file at all, just a free-text objective: open the app, read this task's PRD section, and report visual defects, layout problems, missing elements, console errors, or anything that does not match the requirement.

If either fails, the hook denies the stop with a reason written for the agent: the task, the attempt count, Kane's summary of what failed, and one instruction. Flip the task back to IN_PROGRESS, fix the code, and claim done again. There is no path from "the browser proved this broken" back to "done" that does not go through the code.

Two hook entries in .claude/settings.json are the whole Claude Code side:

{
  "hooks": {
    "Stop": [{ "hooks": [{ "type": "command", "command": ".claude/hooks/guardian-kane-stop.sh" }] }],
    "PostToolUse": [{ "matcher": "Edit|Write", "hooks": [{ "type": "command", "command": ".claude/hooks/guardian-kane-post-tool-use.sh" }] }]
  }
}

PostToolUse quietly records which files each task touched. In this shape of the tool nothing reads that record. Hold on to it; it becomes the most important data in the project later.

Four pairs of builds

To find out whether any of this mattered, I ran four paired experiments. Each pair is the same PRD given to two builds: one unassisted Claude Code agent told "build this", and one going through the full GuardianKane loop. The PRDs were a todo app, a room booking widget, a higher-fidelity booking studio, and ORBITAL: a dense single-page institutional portfolio dashboard with twelve or so interactive subsystems.

ORBITAL is where the chart came from, because it is the only one of the four with enough visual density to have somewhere to hide.

The gated build, orbital-kane, rendered one clean trend line and a visually distinct dashed benchmark, which is what the PRD asked for.

orbital-kane. One series, one line.

The unassisted build, orbital-baseline, drew the doubled line at the top of this post. It also drifted on the alerts panel. The PRD specifies a single-column, full-width, stacked alert list where one alert expands at a time. The baseline built a compact two-column grid instead.

orbital-kane. Full width, stacked, one expanded.

orbital-baseline. Two columns, and the gaps do not match the panels beneath it.

The same screenshot shows a third problem: the space around the alerts block does not match the space around the holdings and risk cards directly below it.

Three defects in one build, and not one of them is visible in a diff. A grid class is not wrong-looking. A chart component that renders one series twice is well formed. The bugs only exist at the moment of render, so the only way to find them is for something to look at the rendered page.

What the gate actually caught

Going in, I assumed the scripted replay was the valuable half of the gate. It is deterministic and repeatable; it is what you would build first. I built the sweep almost as an afterthought and was mildly embarrassed by how fuzzy it was.

The ORBITAL activity log says otherwise. Twelve tasks, all twelve reached KANE_VERIFIED. The sweep flagged something exactly once, on T7, the alerts panel. Both scripted tests for T7 had just passed. They covered expand and switch behavior and asserted nothing about layout. The sweep failed the task anyway and sent it back to IN_PROGRESS. About 43 minutes later the tests passed again and the sweep came back clean.

That log line carries no summary text, so I cannot tell you what the sweep saw. The alerts panel is the same panel the unassisted build got wrong, which is a tempting story. I have no data connecting the two, so I am not telling it.

What I can say is that it was the only failure in the whole build that was a real problem with the app. And the pattern underneath it is general: a scripted test can only check what you thought to assert. The defects that ship are, almost by definition, the ones nobody thought to assert. A strategy made only of assertions written in advance cannot catch the thing most likely to hurt you.

Two things the wiring taught me

Exit codes lie, in both directions. I first mapped kane-cli's exit code straight into the gate. A replay could exit 1 while its own final run_end line reported status: 'passed' with every step passing. A sweep could exit 0, because the objective "completed", while its verdict reported a confirmed bug. The hook now parses the last structured line and trusts the verdict over the process status. If you wire any agentic tool into a gate, read its structured output. The exit code is a summary of a summary.

Killing the process does not kill the browser. kane-cli spawns its own runner, and under it the headless Chrome instances. A timeout that kills the child you spawned leaves those running as orphans, still holding a debugging port. The fix is to spawn detached, so the child leads its own process group, and kill the whole group after every run:

function killProcessGroup(pid) {
  if (!pid) return;
  try {
    process.kill(-pid, 'SIGKILL');
  } catch {
    // ESRCH: group already empty
  }
}

What I take from it

The naive reading is "agents are unreliable, add tests." That was already the advice before I started, so it is not a finding.

The more precise reading is that the failure was never in the code generation. Claude wrote a chart component that was, structurally, fine. The failure was in the self-report. An agent saying "done" is an assertion with no evidence attached, and my first tool had treated that assertion as a state transition.

I have to be careful about what the gated build proves. Its chart was right, but the log does not show the gate catching a doubled chart and forcing a fix. The gated build also had its PRD grilled for ambiguity before any code existed, and twelve tasks each scoped to a PRD section. Which of those made the difference, one pair of builds cannot tell me.

The narrower claim holds. In the gated build, nothing reached "done" on the agent's word. In the unassisted build, "done" was a sentence, and the sentence was wrong.

There is a version of this that is just CI, and for a lot of projects CI is enough. What CI does not give you is a check inside the agent's own loop, before it stops, while it still has the context to fix what it broke. By the time CI goes red, the session is over. The gate fires while the agent is still standing there.

The rest of this series is about the two surprises that followed. The first is where the gated build's advantage actually came from, and it was not the browser. The second is that of the eight ORBITAL tasks that failed a check, seven were not the app's fault at all.

Part 1 of 3. Next: the requirement nobody wrote, on the question that separated the builds and the experiment that found nothing. The code is at github.com/18Abhinav07/adventures-with-kane.

Part 1 of Your Agent Might Actually Lie to You. The original, with the full series navigation, is on my site.