An agent loop fails quietly. It does not throw. It does not log a warning. It makes a plausible, confident, wrong claim at step seven of a twelve-step run, and every step after that inherits the mistake. Nobody notices until the output is already in front of a user, or already merged.
Most agent tooling is built around trusting the agent. quorum is built around the opposite assumption: the agent is going to be wrong sometimes, so something else has to be watching, on every step, in real time.
What it does
quorum sits on top of any agent loop and evaluates each step before letting the run continue. Instead of one judge, it uses five, each checking something specific and independent:
- grounding — is this step supported by what the agent actually retrieved or was given, or is it inventing detail
- consistency — does this step contradict an earlier one in the same run
- safety — does this step attempt something outside the agent's intended scope
- citations — if the step makes a factual claim, is there a real source backing it
- repro — would this step produce the same result if run again, or is it drifting
Each judge returns a confidence score and a vote. quorum aggregates them into a single consensus number for the step. As long as consensus holds, the loop proceeds. The moment it breaks, whether from one judge sharply dissenting or several drifting low together, quorum halts the run and reports exactly which step failed, which judges flagged it, and why, instead of letting the loop finish and hoping someone reviews the output after the fact.
Why halt instead of just log
A log entry is something a human reads later, if they read it at all. A halt is something the pipeline cannot ignore. If an agent loop is doing real work, unattended, the failure mode you actually care about is not "we found out eventually," it is "we found out before anything downstream used the bad step." quorum is built for the second one.
Running it
npx github:rxNxkolai/quorum
Zero runtime dependencies, MIT licensed, and part of a larger toolchain I built for the LLM app lifecycle: tools to author and secure a prompt, test it, ship it, verify what comes out, and supervise the agent that is using it. quorum is the supervision layer, the piece that watches everything else.
The part worth saying out loud
I did not write this alone. I design and direct an AI system that builds production software: I own the spec, the standards, and the verification bar, and a system of agents I run does the implementation work. quorum is a fitting first tool to talk about, because it is a version of the same idea. Nothing in this toolchain ships until it is checked, including the one tool whose entire job is checking.
The full toolchain, with a live demo of quorum catching a hallucination mid-run, is here: rxnxkolai.github.io/portfolio. The code is at github.com/rxNxkolai/quorum.
If you are running agents in production without anything watching them, I would like to know what that actually looks like, and what quorum is still missing for your case.
Top comments (0)