Quorum: What Happens When You Actually Deploy the AI-Safety Tooling You Built

Quorum: what happens when you actually deploy the AI-safety tooling you built

Written for the All Things Agentic Hackathon, Fortified Enterprise Fleet track.

We already build AI-safety tooling. Six separate, independently-tested deterministic tools: a prompt-injection scanner, a claim/provenance integrity engine, a structured human-deliberation protocol, a cross-session drift detector, and an audit logger. Each one worked, on its own, against the thing it was built to test. None of them had ever been asked to do the one job they were actually designed for: watch an autonomous agent, not just review a single chat message.

Quorum is what happens when you point them at that job instead.

The premise

An autonomous coding agent shouldn't be trusted on its own say-so. So Quorum gives one a gate it has to clear before anything it drafts can ship — three of those six tools, wired together as independent, deterministic verifiers, not vibes-based review.

A Worker Agent (Gemini 3.5, via Google's Agent Development Kit, running on Vertex AI) reads a scoped task description, drafts a small code patch, writes a rationale, and lists the specific factual claims that rationale depends on. Before anything leaves the agent, it self-checks its own work — it actually runs the target repository's real test suite against the patch, in a throwaway copy, and only calls itself "done" if that passes.

That proposal then hits the gate:

Sentry scans the agent's own diff and rationale for injected or manipulative content — not the target code, the agent's own output, on the theory that a compromised or confused agent will leave traces in what it writes about its own work.
IntentGraph checks whether the task description is a reformulated return to an objective the gate already rejected — the kind of re-entry attack where you get told no and just ask again with different words.
The Reasoning Kernel independently re-verifies every claim's provenance. It never trusts the agent's self-reported sourcing — it re-reads the cited file itself and only marks a claim VERIFIED if the content actually backs it up.

Every stage writes to an append-only audit trail, win or lose. The gate returns one of three verdicts: PASS, REJECT (which automatically feeds the rejection reason back to the agent for one redraft), or ESCALATE — reserved for exactly the cases a human needs to look at, since one of our six tools (structured deliberation) is deliberately not something code should ever auto-invoke. Earlier adversarial testing on that tool showed an automatic trigger misfiring, so ESCALATE is a genuine dead end for automation by design, not a gap we haven't gotten to.

What "deployed" actually meant

The architecture was solid weeks before the deadline. What wasn't solid — what we hadn't done — was run any of it against real infrastructure. And real infrastructure found bugs that no amount of local testing ever would have.

A proposal whose own tests never ran could still reach PASS. The deployed container didn't have pytest installed — an oversight in the deployment's dependency list, invisible locally because the development environment had it for other reasons. So the Worker Agent's self-check silently failed on every single proposal in production. And the verdict still came back PASS, because none of the three verifiers actually checked whether self-check had passed — they only checked whether the content looked safe. A patch that never successfully proved it worked could still ship. That's fixed now: PASS requires self_check_result.passed, not just clean content, as a hard gate condition, not an afterthought.

A missing binary discarded an already-computed verdict. The action step that opens a pull request on a PASS shells out to git — clone, branch, apply the diff, commit, push. The base container image didn't include the git binary. That's not a bug that produces a wrong answer; it's a bug that produces no answer — the missing-binary failure mode wasn't one our error handling anticipated, so it escaped as a raw, uncaught exception and 500'd the entire request, discarding a verdict that had already been correctly computed a few lines earlier.

Neither of these existed in any test we'd run before deployment. They only existed in the gap between "a laptop's Python environment" and "a fresh Cloud Run container" — two environments that look similar enough to trust and are different enough to hide real bugs.

Proving it, not claiming it

Once the infrastructure gaps were closed, we didn't want to just say the system worked — we wanted to make it prove itself, live, against the actual deployed service, on all three of its own outcomes:

PASS, on a genuinely new task never seen before: Gemini 3.5 reasoned about a real gap in a target ruleset, drafted a fix, self-checked it by actually running pytest inside the container, cleared all three verifiers, and the deployed coordinator opened a real pull request on GitHub — independently confirmed via the GitHub API afterward, not just trusted from the response body.
REJECT, on a crafted manipulation phrase in a diff: Sentry caught it, correctly, on the live service.
ESCALATE, on an ungrounded claim with no supporting evidence: the Reasoning Kernel correctly refused to mark it resolved, and the gate paused for a human instead of guessing.

The audit trail behind all of this is genuinely persisted in Firestore now too — not inferred from the absence of an error, but confirmed by reading the actual documents back out of the database after each call.

What we're not pretending is finished

IntentGraph's embeddings are lexical — an offline hashing vectorizer — not the semantic sentence-transformers model the original design called for. That model has never successfully loaded in any real environment this project has run in, in this sandbox or in the deployed container, so the fallback is the honest current state, not a placeholder waiting to be swapped out. The action currently opens pull requests against a working branch, not main — the right call for a hackathon demo, and the obvious next step before anything broader. And Cloud Run's own cold-start cost — a few seconds fresh, well under a second once warm — hasn't been fully root-caused. We found and fixed one real contributor to cold-start latency during this work; we're not claiming it explains all of it, because it doesn't.

That last habit — say what's proven, say what isn't, and don't blur the line between them — is the actual thesis of this project, not just how we wrote the blog post about it. A gate that only checks whether output looks safe isn't a gate. One that catches its own gaps, live, and discloses them instead of quietly working around them, is closer to what "trustworthy autonomous agent" should actually mean.

Quorum — built for the All Things Agentic Hackathon, Fortified Enterprise Fleet track. Gemini 3.5 via Vertex AI, Google ADK, Cloud Run, Firestore.