Someone asked me the best question after I posted about managing AI agents like a dev team:
And how do you validate quality?
Fair point. If AI ...
For further actions, you may consider blocking this person and/or reporting abuse
Really like the enforced-gates framing — deciding which checks are allowed to stop a release and making them happen every time is the right backbone.
One gate worth adding sits a layer earlier than the validator, and it's easy to miss because every check here inspects an artifact: the validator reads the diff, the review reads the changeset, the chrome_devtool MCP pass drives the real UI. All of them assume the dev agent's report of what it did is true. The failure mode they can't catch is when the agent narrates a tool action that never physically happened — "committed the changes", "created src/foo.ts", "ran the suite, all green" — typed into the turn as prose with no underlying tool call. There's then no artifact for the validator or the E2E pass to fail on, so the orchestration moves on because it trusted the narration.
Since you're already on Claude Code with CLAUDE.md + .claude/agents, the cheap place to put this is a Stop hook (turn-end): before the turn closes, diff the agent's claims against that turn's actual tool_use / tool_result blocks — did "committed" correspond to a real git result with a sha, does the file it said it created exist on disk, did the test command actually execute or was "tests pass" only in the text. If a claimed action has no matching tool return, you hard-fail the turn instead of handing a phantom changeset to gate 4.
We added this after one of our own agents fabricated a tool-output block and a false "empty file" claim inside a turn — the downstream checks had nothing to bite on, because the artifact it described didn't exist. A turn-end check that compares claims to the turn's real tool returns is the missing layer that helps close that gap before the artifact gates even start. Your post-artifact gates and this pre-artifact self-report check are complementary; neither replaces the other.
This is a really good point.
I think you’re right that this is a different layer from the artifact gates I described. My current gates mostly validate things that already exist.
What you’re describing is more like a pre-artifact claim-vs-evidence gate: before trusting the agent’s summary, verify that the claimed action actually has a matching tool result. That distinction is useful. “The code passes validation” and “the agent really performed the action it claims it performed” are not the same thing.
I haven’t implemented this Stop hook check yet, but I like the idea a lot. I have
SessionStartandPreToolUseto inject ENV variables into Claude sessions so it's easy to add also Stop hook checks. This also fits the same principle as the rest of the pipeline: don’t trust narration, trust evidence.Exactly — "don't trust narration, trust evidence" is the whole thing, and since you've already got
SessionStart+PreToolUsewired, the Stop hook is a cheap third leg. Two notes from running ours, in case they save you a step:Compare against the turn's structured
tool_resultentries, never anything in the assistant's prose. The failure that made us build this was an agent typing a fabricated<result>block — a fake tool return — straight into its message, alongside a false "empty file" claim. If the hook scans the assistant text for evidence, that text is exactly where a confabulating turn plants fake evidence. So it has to be assistant-claims (prose) vs the harness's realtool_use/tool_resultrecords — never prose-vs-prose.Map each claim to the return that would have to exist: "committed" → a git result carrying a sha; "created src/foo.ts" → a
Write/Editon that path (we re-stat the file after, too); "tests pass" → a runner result with a real exit code, not the word "pass" in text. No matching return → hard-fail the turn.Honest caveat: it's a tripwire, not a proof. Pattern-based claim extraction has false negatives — a paraphrased claim slips through — and the same agent re-confabulated on a different tool even after we shipped the detector. So it lowers the rate, it doesn't close the hole, which is exactly why your post-artifact gates and this pre-artifact check stay complementary rather than one replacing the other.
Would enjoy seeing how it slots in if you wire it up.
Thanks, this is really useful detail. This goes straight to my backlog to test properly.
When it hits the backlog, the one mapping that bit us hardest:
committedneeds the SHA to resolve, not just be present. We caught a turn that emitted a clean-lookinga3f92c1with no matching Bash result behind it — the confab faked the shape of the evidence, so a check that just greps for a 7-hex string would've waved it through.Two cases worth seeding the test with:
Would genuinely like to hear what slips through when you run it.
The enforced-gates framing is the useful part here. Once AI can produce a lot of plausible code, the real protection is deciding which checks are allowed to stop a release and making them happen every time. I’ve found that one manual end-to-end pass on the core user workflow catches a surprising amount of what unit tests and confident generated code both miss.
Good point, the final end-to-end pass matters.
One thing I’ve started doing is automating part of that with Claude using the
chrome_devtoolMCP. For web services, I can give Claude the core user flows and let it walk through them in the real UI as a smoke test.But I still keep the human pass for the product-level question: does this actually make sense from the user’s perspective?
So for me it is not a replacement for E2E review, but a useful validation layer before the final human check.
I agree. Checking for sense is the main thing. Also, manual testing also allows you to gauge experience, like whether a user journey is frustrating, like no machine could ever tell you. I’ve felt like rage quitting a couple of apps, something an AI is far too polite to do.
Interesting article! Thanks for sharing. Curious how your agents are set up exactly, would you mind sharing? I would like to try to adopt this pattern. Also curious if this works for a team or only individual projects. Would the cross session memory be shared among the whole team or local to each dev?
Thanks Chris (and sorry for my late reply). The setup is mostly a combination of three things:
CLAUDE.mddefines the SDLC flow the agents must follow..claude/agents/contains the role-specific agents, like developer agents andcode-validator..claude/skills/code-review/SKILL.mddefines my custom multi-agent review step.Small naming note: this is my own
code-reviewskill, not Claude’s built-in/code-reviewcommand.The important part is that
CLAUDE.mdmakes the process enforceable. After development, thecode-validatormust return PASS. Only then the workflow invokes mycode-reviewskill on the full changeset. If that review finds real issues, the relevant dev agent fixes them,code-validatorruns again, and the fix is committed.Inside the review skill I currently run four parallel review agents:
The biggest lesson for me was to make the review very strict about what it is allowed to report. It should flag compile errors, clear logic bugs, and unambiguous architecture violations with exact rule citations. It should not report style opinions, vague “could be better” suggestions, pre-existing issues, or things linters already catch.
For adoption, I would not start with the full version. I would start with the smallest useful loop:
Developer agent → code-validator → commit only after PASS.
Then add the full multi-agent review before PR creation.
For teams, I think the pattern works even better, but the shared knowledge should live in versioned project files: architecture docs,
CLAUDE.md, agent / skills definitions, CI scripts, and validation rules. Personal/local memory is useful for one developer, but team-level memory should become repo-level knowledge as soon as possible. And for teams I would recommend to check Claude Marketplace GitHub repo setup, it could solve many maintainability issues with agents / skills.your 8-gate pipeline is wild lol, way more rigor than most teams even bother with. does the multi-agent review step ever actually catch stuff the tests miss, or is it more insurance at this point?
Both at first. Six months in, it is genuinely producing value.
The reason is that the two layers answer different questions. Tests tell me the code does what the code was written to do. The review agents check the implementation against acceptance criteria that were written before anyone implemented anything. That gap is where they find things tests structurally cannot.
Real case from a "lock navigation while the pipeline is running" task:
LINT=PASS
TESTS=6104 passed, 0 failed
COVERAGE=94.6%
RESULT=FAIL
The route guard depended on router context the production app never supplied. The unit test fabricated that context, so AI agent verified a callback under conditions that did not exist in the real app. The validator agent proved it by deleting the guard and rerunning the suite. All 6,104 tests still passed.
Green suite. Feature that never worked.
It also changed what I do on the PR. I do not read the diff line by line anymore. I read the artifacts the run produced: which acceptance criterion was verified, with what evidence, what got escalated to me. I approve on that.
I am writing a follow-up article on exactly this mechanism, what stops AI validation from becoming an expensive rubber stamp. Happy to ping you here when it is out.