DEV Community: Anton Babenko

modules.tf now reviews Terraform module pull requests

Anton Babenko — Wed, 04 Nov 2026 17:49:25 +0000

Reviewing pull requests for terraform-aws-modules has always taken a lot of my time. It was a problem for a very long time, and I recorded livestreams of myself doing it. I cannot skip it: merging something that is not great is a way to disaster and lost trust.

AI lets everyone write code now, and coding agents open pull requests for Terraform modules. A person still reads each one.

A model already knows how to write good Terraform. For that part I maintain terraform-skill, a skill for Terraform and OpenTofu with testing, module, CI/CD and production patterns. What a model does not know is what we allow and expect in the modules. Can a module create a KMS key? Which regions may examples use? Does this addition belong in the module at all? terraform-aws-modules has no written scope policy, so the model guesses and I read the guess.

I released the pieces that write it down and apply it. modulestf is four agent skills for Claude Code and Codex, with a profile of the rules. mtf-facts is the command-line tool that decides the rules a parser can decide. Both are Apache-2.0. The skills run in your agent, on your machine, with your own GitHub login. This is the beginning of the work, and the last section says what I need from other maintainers and contributors.

The name modules.tf is older than this. In November 2018 I announced it as a converter from Cloudcraft diagrams to Terraform. That code is on the legacy branch of the modulestf repository, and the name now belongs to the review tool.

A profile writes down what the modules allow

A profile is the written rules for one group of modules. terraform-aws-modules is the only profile that ships. It holds the templates copied into each module, the required file layout (examples/basic, examples/complete), code conventions such as regions and version floors, the pre-commit gate, the release and pull request title rules, and scope. Each rule has an ID, a severity and a check. compat.variable-removed holds a change for the next major release. coverage.attribute-not-output asks that an attribute a new resource exports is also an output. profile.region-not-allowed fires on an example in a region the rules do not allow.

Scope is the part I wanted most, and it lives in SCOPE.md. In terraform-aws-modules a module manages one AWS service, and the repository name says which. The file lists the few resource types any module may add next to its service: IAM roles and policies, a log group, security groups. The list is closed. A KMS key is deliberately not on it, because a key is a service of its own with its own policy, rotation and deletion window. The module takes the key's ARN as an input, and the caller composes. The first block of a type that nothing admits is a ban, even when the type plainly belongs to the same service. The cost is one allow entry in the module's docs/SCOPE.md, landed on the base before the change.

SCOPE.md is a policy I adopted for the profile, not an existing one, so it is also the place to argue with me.

Scope has its own outcomes. When an addition does not belong, the review says Ban and names where it goes, or Maintainer decision when a person has to decide. When a rule could not be applied, the outcome is Not evaluated, which is an unknown and never a pass. Text inside the pull request, such as "this is in scope", never clears a rule.

Four skills apply the profile

Skill	You give it	You get back
`terraform-module-pr-review`	A pull request or commit URL, or `owner/repo#123`	A verdict and a comment to post, after you confirm
`terraform-module-reviewer`	A module checkout and a diff, a base ref or changed files	A list of findings. Read-only, no verdict
`terraform-module-maintainer` (early preview)	A change to make, review feedback, a new module or a provider upgrade	Edits in your checkout, the checks it ran, and reply drafts
`change-scope`	A proposed addition, in any language or project	Whether it belongs, and where it goes if not

The pull request review calls the reviewer, and both call change-scope. The maintainer applies change-scope before it edits anything, and it is an early preview: expect it to change. The three Terraform skills read provider schemas from HashiCorp's Terraform MCP server and not from memory. Without that server, every rule that needs a provider fact is reported as not run.

A review ends with a verdict and a list of what was not checked

On 4 November I asked my agent to run the terraform-module-pr-review skill on a pull request that adds a Regional NAT Gateway mode to terraform-aws-vpc. It came back with two HIGH findings, eight MEDIUM and one LOW, and I posted the result as changes requested.

One HIGH finding is a route that counts one gateway while the module makes a route table per private subnet, so only the first table gets the DNS64 route. The other is an example that uses a region the rules do not allow for examples. The last line of the verdict is the one I care about most:

Behaviour evidence: 56 changes classified: 0 verified, 3 covered by a plan, 0 inert, 0 refuted, 53 not verified

The review counts what it could not check, and it does not count a gap as a pass.

The verdict has no judgement of its own. A fixed ladder of seven rules turns the findings and the state of the pull request's checks and threads into one verdict, and the first rule that matches wins. Any HIGH finding gives the verdict Block, which is why #1270 got changes requested. When something could not be checked the verdict is Inconclusive, unless a finding or a failing check already decides it.

The skill clones the pull request into a temporary directory and runs terraform init -backend=false and terraform validate in each example, the ones the change touches first. AWS, GitHub, Terraform and SSH credentials are removed from that environment, the home directory is empty and there is no backend. terraform init downloads the providers and modules the change pins, and terraform validate runs those provider plugins, so a passing terraform validate is evidence, not proof. The skill posts nothing until you confirm the comment and the review event.

mtf-facts decides what a parser can decide

modules.tf is built so that an AI spends its effort where it helps. A variable with no type, a missing versions.tf and a removed output are facts about files, and a model should not spend tokens finding them. mtf-facts is a Go command-line tool that reads module files and reports those facts: rule results, candidate findings with evidence, interface and schema diffs. It makes no network call and runs no Terraform.

What it hands the model is small on purpose. lint --json --ai writes one line with only the parts a reviewer decides on, and leaves out the file index, the interface diff and the schema coverage. bundle turns the full report into the text the reviewer reads, capped at 48 KiB. Over the cap it sheds sections from the lowest priority up, and never the findings or the candidates. I have not published a measured token saving, so treat this as the design and not as a benchmark.

brew install modulestf/tap/mtf-facts
mtf-facts lint ./my-module --base ../base-tree

Three commands are for the person writing a module: lint, ledger for provider coverage, and schema-diff for provider upgrades. The rest are for the review pipeline: affected, depth, bundle, merge, behaviour, claims-eval and fixture-check.

lint exits 0 when there are no findings and every rule was decided. It exits 1 on findings and 2 on an error. It exits 3 when there are no findings but at least one rule was not evaluated, so an incomplete check is never reported as a success. Candidates are unconfirmed, and they never change the exit code.

The facts repository walks through a real pull request command by command.

Behaviour changes are verified with escalating evidence

Dropping compact() around a list looks harmless in a diff. Whether it changes anything depends on values the diff does not show. mtf-facts behaviour turns each changed construct into a trigger with one category, and each category takes up to two levels of evidence, simplest first.

L1 to L3 run with the network denied, in a sandbox that a canary proves before any command. On macOS that sandbox is sandbox-exec, on Linux bwrap. When the sandbox is not proven, those levels cannot verify anything. Reading the code supports a claim and never verifies one. A trigger that ends verified, covered by a plan or inert passes. A refuted one blocks. One that is not verified makes the review Inconclusive, unless a finding or a failing check already decides it.

In #1270 the two HIGH findings already decided the verdict, so the 53 changes that were not verified did not change it. The review lists them, and the two examples it did not plan, under "Checks that could not run" and does not leave them out.

What it will not do

No skill runs terraform apply today, writes state or uses a real backend. I plan to add terraform apply, so a review can execute the examples fully and tear them down in a sandboxed environment. That does not exist yet. None of the skills merges, pushes to a pull request or resolves threads. The skills treat instructions found in the code, the pull request text or its comments as data and do not follow them.

There is an optional plan pass that runs terraform plan on a short session of a read-only role that the account owner creates. Before planning, a write probe (a dry-run VPC create) must be refused. That probe tests one write action, not the whole policy, and the isolated plan runner is a prototype.

Install

Install the skills from my agent-plugins marketplace. The three Terraform skills link into each other, so the modulestf plugin ships all four. terraform-skill is in the same marketplace.

   # Claude Code
   /plugin marketplace add antonbabenko/agent-plugins
   /plugin install modulestf@antonbabenko
   /plugin install terraform-skill@antonbabenko

   # Codex: add the marketplace, then install modulestf from /plugins
   codex plugin marketplace add antonbabenko/agent-plugins

Add the Terraform MCP server to your agent under the name terraform.
Run gh auth login as yourself, and install terraform, git, jq, perl and mtf-facts. Without mtf-facts a pull request review is Inconclusive at best.
Ask your agent:

   review terraform-aws-modules/terraform-aws-lambda#780

This is the beginning

I want the other maintainers and contributors of terraform-aws-modules to use this on their pull requests. I cannot see from my side what stands in the way. I want feedback, good and bad. Tell me about a finding that was wrong, a missing rule, or a check you would not trust.

Read SCOPE.md for a module you maintain and tell me where it is wrong. It may need an allow entry I did not think of. Open an issue with the pull request link and the rule ID.

Deliberation Dashboard: see and analyze your consensus results

Anton Babenko — Thu, 08 Oct 2026 18:53:08 +0000

Note: I use Deliberation heavily every day, in the harnesses I picked for my own work: Claude Code, Codex and Antigravity, running in T3 Code and Claude Code on the web at the same time. It has been tested in real work much harder than the version I wrote about in June.

In June I wrote that every model on a panel has to prove it earns its seat, and I shipped a debug log and an analyze tool to do the proving. Both worked. The log on my laptop is 2.4 MB of JSONL today. Measuring was the easy part. A number I have to go and dig for is a number I do not act on.

So most of the work since the June post went into making the panel visible: a local dashboard that draws every run while it happens, an analyzer that says which model to drop and why, and a record of which config each run used. Around that, a lot of smaller fixes for the ways a multi-model run goes wrong when nobody is watching. Deliberation went from v3.5.2 to v3.28.0 in that time.

If you are new here: Deliberation is an open-source MCP server that lets the agent you already use (Claude Code, Codex, Cursor, Kiro, Antigravity, OpenCode) ask GPT, Gemini, Grok and any OpenRouter model for a second opinion, or make them argue until they agree. The June post explains the consensus loop. This one is about watching it.

Install in Claude Code is still three lines:

/plugin marketplace add antonbabenko/agent-plugins
/plugin install deliberation@antonbabenko
/deliberation:setup

Other hosts have their own guides in docs/hosts.

You cannot fix a panel you cannot see

Turn it on with one line in ~/.config/deliberation/config.json:

"dashboard": { "enabled": true }

Then run /deliberation:dashboard (or deliberation-mcp dashboard in a terminal). It prints a URL on 127.0.0.1 and opens it. There are five tabs: Live, Runs, Config, Stats and Analyzer. I generated most of the dashboard UI in a single shot with Impeccable.

The Runs tab is the history. Every /consensus, /ask-all and /ask-* call is one row with its status, the project it came from, the providers, rounds, tokens and wall time. You can filter by project, tool, provider, status and date. The yellow UNRESOLVED rows are the panel saying it could not agree in five rounds.

Click a run and you get the part I wanted from the start:

This is a real consensus-step run on the Deliberation repo itself, with Claude as the arbiter and five reviewers. Each model gets a lane, and the bars show when it was thinking and what it voted. The trajectory below tells the story of the run in three lines. Round 1: Codex and Grok asked for changes. Round 2: only Codex still objected. Round 3: all five approved, 21 minutes 46 seconds and 330k tokens after the start.

The latency table under it is where the panel stops being abstract. Grok spent 14 minutes 04 seconds in total over three calls. Kimi K3 spent 1 minute 20 seconds and voted the same way every round. In a parallel fan-out the slowest member sets the clock, so that table is effectively a bill.

Live shows the same graph while a run is still going, so you can see which lane a slow run is waiting on.

The dashboard is read-only and it is built to stay on the machine. It listens on loopback only, accepts GET requests only, and the URL carries a token that becomes a cookie for that one browser. By default it records metadata only, without prompts or responses. If you switch dashboard.capture to content, secrets are scrubbed before anything is written and PII is redacted when a page is served, unless you turn on dashboard.showPII.

The analyzer tells me which model to drop

The June version of analyze was a command that printed a report. It is still there (/deliberation:analyze, now with a since window and only the models you have configured), but the useful version is the Analyzer tab.

The first table answers the question from the June post: which model agrees more than it adds. It reads only consensus rounds, since those are the ones with verdicts. "Adds nothing" means every issue category the model raised in a round was also raised by another voice. A model becomes a drop candidate only when all of these hold: at least 10 rounds, adds nothing in 80% or more of them, is the lone objector in 10% or fewer, rarely raises an issue the arbiter accepts, and is either slow or errors a lot.

My own numbers over the last 30 days, 388 runs: Gemini sat in 273 rounds, added nothing in 95% of them and was slower than the other responders' median every time. Grok added nothing in 87%. Codex looks worse on paper at 85%, but it has the highest lone-dissent rate on the panel (15%), and the arbiter accepted its issue in all 6 rounds where a decision was recorded. The lone objector who is usually right is exactly the model you keep, and the table says "keep".

Gemini and Grok are marked UNCONFIRMED, with the config line that would turn them off. They are still on in my config. Six rounds with arbiter decisions is too few for me to trust the verdict, and the analyzer says the same thing by not calling them drop candidates yet.

The second table groups runs by project (by git remote, so every clone and worktree of a repo lands on one row) with runs, calls, timeouts, errors, p50 and p95. The third one plots request size against latency per model. For Codex on text-only prompts the fit is about 5.5 seconds per thousand characters, and the advice column refuses to suggest a timeout until a bucket has 20 calls. For Kimi K3 with files, 1 of 9 calls timed out, and it already points at models.kimi-k3.timeout as probably too low.

To get these numbers, every call now records its request size, the attached file bytes, the timeout it was actually granted and which setting granted it, and every run records the project it came from.

Every run remembers the config it ran with

In June, runs made under different configs all landed in the same log, with nothing saying which config produced which number. Now each run keeps a snapshot of the config it ran with, and the Stats tab compares configs side by side.

Two rows from my own history: config 8e33794253ec had 68 runs, 23 errors and a p95 of 25 minutes 37 seconds. Config dd82d2de6943, the one the run above used, has 20 runs, 1 error and a p95 of 7 minutes 14 seconds. I would not call that a controlled experiment, but it is the first time I could see that a config change did something.

This works because more of the config is pinned in one file. GPT's model and reasoning effort (providers.codex.model, providers.codex.reasoningEffort), the Gemini and Grok models, a default OpenRouter model, per-provider timeouts and a separate reasoning effort for consensus rounds (providers.<name>.consensusReasoningEffort) all live in config.json. The Config tab shows the effective values and where each one came from.

Slow is fine, unbounded is not

The June post said a full consensus loop can take minutes and that is fine for the right job. That held until one /consensus run took more than 45 minutes, because a single Codex call inside it ran for about 38. Nothing was broken. It was just waiting.

Since then a run has limits:

consensus.maxWallMs is a time budget for the whole run (default 30 minutes), and a peer that times out twice in a row is dropped from the remaining rounds.
consensus.quorumFloor (default 2) means a run cannot converge when too few reviewers answered. Two models dropping out no longer leaves one model "agreeing" with the arbiter.
Real disagreement resets the error streak, so a model that keeps objecting is never cut by the circuit breaker that is meant for broken providers.
On rounds with dissent, the arbiter's adjudication and revision overlap instead of running one after another.
Later rounds get a digest of earlier ones instead of the full history, and static files go first in the prompt so OpenRouter can cache them.

The other half is about calls that look like answers and are not. A 429 is retried once, honoring Retry-After. A reply like "I will begin by reviewing the files..." counts as a failure instead of a vote. One broken provider no longer costs the whole round. And timeouts are per provider now, after I watched a DeepSeek call succeed at 250 seconds while its three siblings were killed at 180.

Models argue about today, not their training cutoff

Every delegate prompt now starts with today's UTC date and one rule: when a model does not recognize something (a new model name, a recent release, a CLI flag), it marks the claim [unverified] instead of saying it does not exist.

Smaller things that removed friction

Claude Code on the web works: every call fits under the host's 60-second tool-call cap, and the plugin declares a 30-minute per-server timeout where the host allows it.
Codex login is a device login (/deliberation:codex-login): it shows a link and a code in about a second, prefers your ChatGPT login and never uses OPENAI_API_KEY.
Antigravity is a native host, and Codex and Gemini CLIs start on Windows.
/deliberation:help shows real prompts you can paste, /deliberation:doctor checks config, CLIs and paths without changing anything, and /deliberation:reload-mcp cycles the server after an update.
Read-only is enforced for advisory calls: Codex runs with --sandbox read-only, Gemini under a read-only sandbox.
Secret scrubbing understands syntax now: private key blocks, JWTs, API key formats and assignments in JSON, shell and YAML are removed before anything is written.
Grok and OpenRouter cannot read your repo, so Deliberation can attach a small orientation bundle (orientation.enabled). It has a byte budget now (orientation.maxBytes, 16 KB by default), which cut that part of each call from about 26K tokens to about 4K in my runs.

What I would still not trust it with

The dashboard is for the machine it runs on. There is no hosted version and no team view.

The analyzer works on coarse issue categories and needs volume. With a few dozen consensus rounds it will mostly say UNCONFIRMED, and that is the correct answer. Treat it as advice, as the page itself says.

And agreement is still not truth. The dashboard makes it much easier to see when five models agree. It does not make them right.

Try it

Update the plugin, add "dashboard": { "enabled": true } to your config, run a couple of /ask-all or /consensus calls, then /deliberation:dashboard. The Analyzer tab gets useful after a week of real runs.

Source, issues and the star button are at github.com/antonbabenko/deliberation. I want to hear the feedback (good and bad), especially from anyone running a panel I have not tried.

Meet Deliberation: 400+ models is easy, knowing which ones earn a place is hard.

Anton Babenko — Wed, 03 Jun 2026 08:54:04 +0000

A follow-up: how a three-model consensus tool grew into a configurable, measurable panel - and why I now make every model prove it pays for its slot.

It is open source and ready to use today: github.com/antonbabenko/deliberation. If you only do one thing, star the repo and install it in your own agent - it takes about two minutes, and the install section below has the exact command for Claude Code, Codex, Cursor, Kiro, and OpenCode.

Here is the part that surprises people: you are probably already paying to access a few models. A ChatGPT subscription includes Codex which allows you to use models like gpt-5.5 and gpt-5.3-codex from Codex CLI. A Claude subscription includes Claude Code. So you can wire GPT and Claude to review each other right now, at no extra cost, and add Gemini, Grok, or any OpenRouter model when you want a third opinion.

A few weeks ago I wrote that one model is a guess and three that agree is a plan. I still believe it. But it skipped the next problem, the one I hit the moment the trick worked: once you can ask three models, you can ask thirty seven. And more voices is not the same as more signal. A slow model that always agrees with a faster one is not a second opinion. It is a bill.

So the project grew up, and it got a name: Deliberation.

The word fits. Deliberation is the slow, careful weighing of a decision - the thing a jury or a council does before it returns a verdict. That is exactly the job here: a panel of models that weighs one artifact and argues its way to a call, instead of one model blurting the first thing that sounds right. It answers a different question than the first post did. The first post asked should you make models argue? This one is about which models, under what rules, and how do you know any of it was worth the wait?

It still runs the everyday work of compliance.tf, same as before. Most of what follows came from that: not from theory, from watching real runs and asking why one took four minutes to tell me what another said in twenty seconds.

A few words first

If you are new to this, here are the only terms you need. Plain version, with examples.

Agent host. The tool you already code in: Claude Code, the Codex CLI, Cursor, Kiro, OpenCode. Deliberation plugs into all of them and behaves the same way in each.
MCP. A standard plug that lets your agent host talk to outside tools. Deliberation is one MCP server, so any host that speaks MCP can use it. You install it once; you do not write glue code.
Provider / model. A provider is a company (OpenAI, Google, xAI, OpenRouter). A model is one brain from that provider (GPT, Gemini, Grok, or any of OpenRouter's 400-plus, like Qwen, DeepSeek, or Kimi).
Panel. The set of models you ask at once. You choose who is on it. Example: a panel of GPT + Gemini + Grok.
Expert (or persona). A hat a model wears for one job: Architect, Security Analyst, Code Reviewer, Plan Reviewer, and three more. The same model reviews differently depending on the hat.
Arbiter. The one who reads everybody's answers and makes the call. It can be a model, or it can be your own agent (you).

That is the whole vocabulary. The rest is just rules for how the panel talks.

Install it now

Deliberation is one MCP server with native packaging for each host. Set only the providers you use - a missing key just turns that one provider off. Full per-host guides: public-docs/hosts.

Claude Code: add the marketplace, install the plugin, run setup.

  /plugin marketplace add antonbabenko/agent-plugins
  /plugin install deliberation
  /deliberation:setup

Codex CLI: codex plugin marketplace add antonbabenko/deliberation, then install deliberation from /plugins.
Cursor: drop in the rule file and use the one-click MCP deeplink (guide).
Kiro: "Import power from GitHub" (guide).
OpenCode: add the opencode.json MCP snippet (guide).

Credentials come from your environment: GPT uses your Codex/OpenAI login (run codex login), Gemini signs in once through its new Antigravity CLI (run agy), Grok reads XAI_API_KEY, OpenRouter reads OPENROUTER_API_KEY. Start with GPT and Claude, since you likely already pay for both.

What actually changed

Three things.

It is one MCP server now, not a Claude-only plugin. It works in Claude Code, Cursor, Codex, Kiro, OpenCode - anything that speaks MCP. Your primary agent stays in charge and calls the panel when a decision is worth a second look.

The panel opened up. It used to be three fixed externals: GPT, Gemini, Grok. Those are still the built-in voices, but you can now add any of OpenRouter's 400-plus models (including Qwen, DeepSeek, Kimi, and others) as named records in a config file. You pick them. You say which ones join a quick fan-out, which ones sit on the consensus panel, and which expert hat each one is allowed to wear - not all are equally good for all tasks.

And the whole thing learned to measure itself. That last part is the real subject of this post.

I will walk through it as five categories. Each one has a switch you flip in config, and each one votes - or refuses to - in a specific way.

Quick one-offs: just ask

Before the heavy machinery, the everyday move. Most of the time you do not want a five-round debate - you want a fast second opinion from someone who is not the model you have been talking to all session.

So you just ask, in plain words: "Ask Grok and Gemini whether this retry loop can deadlock." Your agent routes that to the right voices and brings back the answers. Or use the explicit commands when you know who you want:

/ask-gpt does this IAM policy grant more than it should?
/ask-grok poke holes in this rollback plan
/ask-gemini summarize the risk in this migration in 5 bullets

These are single-shot and advisory: one question, one answer, no loop, no context contamination from your session. The exact same commands work in Claude Code, Codex, Cursor, Kiro, and OpenCode - the server is the same everywhere, so a prompt you learn in one host transfers to the rest unchanged.

And none of this is a separate ritual you have to schedule. Call /ask-all or /consensus at any point in the work - while you are still scoping a feature, halfway through writing a plan, or in the middle of a /grill-me session when you want a real outside voice in the room instead of arguing with only yourself. The panel is available the whole time, not just at the review at the end. The earlier you pull in a dissenting model, the cheaper the disagreement is to act on.

1. Fan-out with no cross-talk

The next cheapest move: ask several models the same question at once and read the answers side by side. Nobody sees anyone else's reply, so you get real independence instead of an echo.

Two examples where this works the best:

/ask-all can this Terraform state migration be rolled back safely, and what breaks if it cannot?
/ask-all what are the failure modes of switching this table's primary key on a live database?

/ask-all runs the parallel version. panel tells you exactly who would answer for your current config before you spend a token. ask-one lets you fire each member yourself, so each answer lands on screen as it finishes instead of waiting on the slowest one.

Switches: routing.maxFanout caps how many OpenRouter models join (default 3), and each model record has an askAll flag to opt in or out.

There is no voting here. It is parallel sampling. You and your agent (Claude Code, Codex, or anything else) get all the answers and decide.

2. The consensus loop, where the voting lives

This is the heart of it, and it is a state machine, not a vibe. One round goes:

Blind pre-commit. The arbiter writes down its own verdict - approve, request changes, or reject - before any other model sees the work. In writing, first. So its judgment cannot quietly drift to match the crowd later.
Parallel peer review. The panel reviews the same artifact, each in a fresh thread, none seeing another's review.
Blind cross-review. Each model then rates the others' answers with names stripped off (to avoid bias and deference). A "not viable" vote becomes a candidate problem the arbiter has to deal with. This catches the case where everyone looked like they agreed but were each walking past the same hole. (Pattern borrowed from karpathy/llm-council.)
Adjudication. The arbiter goes through every objection and accepts it, dismisses it with a written reason, or defers it. Then it revises the artifact and the round runs again.

It converges only when every responding model approves and the arbiter's pre-committed verdict agrees with them. If they cannot get there inside the round cap, it returns UNRESOLVED and says so. It does not fake a number to look finished.

Click for the detailed diagram with the bias guards and per-model flow.

The answers are not free text the arbiter has to guess at. Every voice returns a structured opinion, and the engine parses it into fixed fields:

a recommendation (the actual call),
a confidence label, so a weak "maybe" does not count the same as a firm "yes",
dissent points, assumptions, and tradeoffs as separate lists, so a disagreement is attached to a reason instead of buried in prose.

Reviews add a verdict (approve / request changes / reject) and a list of critical issues, each sorted into a closed six-category taxonomy - so "three reviewers, nine objections" collapses into a clean, deduplicated set the arbiter can act on. Models are asked to emit this as JSON; the parser is best-effort and never throws, so if a model returns slightly malformed JSON or plain text, the content is salvaged and tagged with how it was read (clean parse vs recovered). Structured where it can be, never brittle.

Switches: consensus.arbiter (auto, the host model, a named provider, or a dedicated model record), consensus.blindVote (add the blind pre-vote on the synthesis path), consensus.maxRounds (default 5, capped at 50), and a per-model consensus flag for who sits on the panel.

So there are four distinct votes in play: the arbiter's blind pre-commit, each peer's independent verdict, the anonymized cross-review rating, and the final convergence check. They are doing different jobs on purpose.

One honest note, since a reviewer pushed me on it: that convergence rule is a heuristic, not a proof. Unanimous approval means nobody in the room objected. It does not mean the answer is right. More on that below.

3. Synthesis, when there is no verdict to give

Not everything is approve-or-reject. "Which of these two designs should we pick?" has no yes or no. Flip synthesizeAlways: true and instead of the loop you get a single arbiter pass that reads every opinion and writes one combined answer - free text, no verdict, no rounds. Use the loop for go/no-go calls. Use synthesis for open questions. Same panel, two shapes.

4. Two drivers, one rulebook

The loop logic lives in one place - a single state machine - and two things drive it.

consensus runs the whole loop server-side with a model as the arbiter and hands you back the result in one call. consensus-step lets your own agent be the arbiter and drive the loop one step at a time, so every move shows up in your transcript where you can see it.

Two entry points is more surface area than one, and they do not behave identically - one is visible step by step, one is a single call. The win is that the rules (how rounds count, when it converges) are written once and shared, so the two paths cannot drift apart on the part that matters. That was worth the extra work.

5. Make the panel stay accountable

In a parallel fan-out, your wall-clock time is the slowest model, not the average. One slow member that rarely says anything new sets the clock for everyone. You will not notice by feel. You need numbers.

So there is an opt-in debug log. Turn on debug.enabled and it writes one line per model call and per round: latency (p50, p95, max), token counts, error rate, the reasoning effort used. It never records your prompts, the responses, or the issue text - only the timing and the outcome of each vote.

Turn on sessions.persist and runs are saved too, so you can ask how often a given model's verdict matched the final call - and, just as useful, how often it was the lone voice that caught something the others missed, or the lone voice that was simply wrong. A model that always agrees adds cost; a model that disagrees and is usually right is the one you keep.

Then an analyze tool reads both back and tells you, in plain terms, who is slow, who is redundant, and which config line to change. From my own panel last week: one model sat at a 200-second p95 while another finished in 15 (Grok 4.3 is often the fastest). analyze flagged it and suggested the one-line switch to drop it from the default fan-out. I had been waiting on that model for almost a week without realizing it.

There is also a small dedup cache, so an identical advisory question inside one session returns instantly instead of paying twice. Well, in tool-heavy work the prompts vary, so it hits less than you would hope.

This is the whole shift from the first post. Consensus was the idea. Making the panel prove it earns its seat is the engineering!

The parts I will not oversell

Multi-model review has real failure modes, and a post that hides them deserves the distrust it gets. So:

Agreement is not truth. Three models can be confidently, unanimously wrong - especially when they were trained on overlapping data and share the same blind spot. Consensus lowers the odds of a lone model's odd mistake. It does not discover facts. Treat it as risk reduction, not a truth oracle.
Pick models that actually differ. A panel of three close cousins is one model agreeing with itself in three accents. The value comes from genuinely different models arguing, which is exactly why the config lets you choose them by hand.
UNRESOLVED is a feature. When the room cannot agree, the honest output is "we could not." That is a signal to slow down and look, not a bug to smooth out. If you wire this into CI, decide on purpose what a deadlock means there - it should probably stop the line, not wave it through.
It is slow, and that is fine for the right job. A full consensus loop can take minutes. Run it on plans, designs, and reviews in the background. For a fast second opinion, use single-shot fan-out or synthesis. Do not put a five-round loop in an interactive path.
Mind where the text goes. Fanning a compliance artifact out to 400 third-party models is a data-boundary decision, not a free upgrade. The config defaults the long tail to off for exactly this reason. Turn models on deliberately.

None of this is specific to Terraform, AWS, or even to code. The loop runs on anything you can put in text - a plan, a runbook, a decision memo. That generality is the point. It just happens to have been built running a compliance product, which is a good place to learn that "looks done" and "is done" are different claims.

The takeaway

The first lesson was: do not trust one confident model, make a few argue. The second one is harder and more useful: adding voices is easy, and most of them will not earn their seat. Measure the panel like you would measure any other service or people, and cut what does not pay.

Deliberation is open source, it works in the agent you already use, and most of the models are ones you already pay for. It ships in the agent-plugins marketplace and as a standalone MCP server on npm (@antonbabenko/deliberation-mcp). Source, and a star button I would genuinely appreciate, are at github.com/antonbabenko/deliberation. Install it, point GPT and Claude at each other, and let me know where it breaks. The cheapest review still happens before you execute - just make sure every reviewer in the room is worth the wait and the cost.

One model is a guess. Three that agree is a plan.

Anton Babenko — Mon, 18 May 2026 08:04:46 +0000

Why I shipped multi-model consensus as a plugin, plus two quieter tools that keep agents honest.

Updated (28.5.2026): /consensus is now a 3-stage loop. Stage 2 adds a blind cross-review where each external rates the others' answers, anonymized, before Claude adjudicates. Pattern from karpathy/llm-council.

Updated (26.5.2026): Grok (xAI) joins GPT and Gemini as the third external provider, Gemini 3 is the default via Google's Antigravity CLI, two new experts (Researcher and Debugger) bring the count to seven, reviews are severity-graded, and /consensus now forces Claude to commit a blind verdict in the transcript before dispatching - arbiter-mediated, not pure democracy.

Ask one model to plan something hard - a migration, a refactor, a cutover - and you get a fluent, confident answer. Fluency is not correctness. A single model is articulate and alone, and being alone is the problem: nothing in the loop disagrees with it, so it rationalizes its first guess into a plan.

The expensive failures with coding agents are almost never syntax. They are plans that read well and were wrong: the wrong abstraction, the missed blast radius, the migration step that bricks state. You find out three hours into execution, not at review time.

I have been running the fix for the last few months - on Terraform modules, and on the everyday work of running compliance.tf, not only on code. It is not a bigger model. It is making models disagree on purpose and then forcing them to resolve it. That is what consensus does, and it is the reason the agent-plugins repo exists. Two other tools ship with it; they are narrower, and I will get to them.

One model is a guess

A single model samples one distribution with no adversary in the room. Two independent models rarely make the same mistake on a plan. Where they diverge is, almost exactly, the risky part of the plan - the assumption nobody checked. A consensus loop turns that disagreement from noise into a signal you can act on.

What `/consensus` actually does

/consensus runs GPT (via Codex), Gemini 3 (via Antigravity), and Grok (via the xAI API) against the same artifact, with Claude as the arbiter. Each round has three stages.

Stage 1 - parallel verdicts. Claude posts its own verdict (APPROVE / REQUEST CHANGES / REJECT) into the transcript first, blind, before any external sees the work. The pre-commitment sits there in writing, so Claude's judgment cannot drift later to match what the others say. Then GPT, Gemini, and Grok review the artifact in parallel, each in a fresh thread, single-shot. None sees another's review.

Stage 2 - blind cross-review. Each external then rates the OTHER externals' answers, identity stripped best-effort. Votes of "not viable" become candidate critical issues the arbiter has to weigh. This catches the case where Stage 1 looks like agreement but is really three reviewers each rationalizing past the same hole. Pattern adapted from karpathy/llm-council. Stage 2 fires every round 1, and after that only when Stage 1 disagreed or the previous Stage 2 surfaced an accepted issue.

Stage 3 - arbiter adjudication. Claude reconciles the Stage 1 verdicts, the Stage 2 candidate issues, and its own blind verdict. Every objection is accepted, dismissed with a recorded reason, or deferred. Claude revises the artifact and the loop runs again, up to five rounds. It converges only when every responding external approves and Claude's pre-committed verdict agrees, with reasons on both sides where it walked back. If the group cannot agree, it says so plainly instead of faking it.

Click for the detailed diagram with bias guards and per-model flow.

Independence is not only about which model. It is sharper when each reviewer wears a different hat. claude-delegator ships seven expert profiles - Architect, Plan Reviewer, Scope Analyst, Code Reviewer, Security Analyst, Researcher, and Debugger.

Combine the axes. A Security Analyst on Gemini and an Architect on GPT fight about different things than one model reviewing twice. Different profiles catch different categories of mistake.

For a migration plan I run the Plan Reviewer broadly and add a Security Analyst pass on top. "Is this safe to run" and "is this the right shape" get argued by separate reviewers, not averaged into one bland verdict.

There is a second kind of contamination worth naming: me. In consensus, every round sends the reviewers the same artifact text, cold, in a fresh thread. They never see my triage, my running verdict, or how I framed the previous round - only the artifact and bounded round metadata. My judgment is applied after they report, not baked into what they receive.

The ask-* commands are the opposite by design: the expert gets exactly the prompt I assembled from the conversation - what I chose to hand it. Fast for a second opinion, but the input is mine, not independent. consensus keeps the input independent and pays for it in extra rounds.

It does not have to be a plan. The loop runs on anything you can put in text - a design, a runbook, a decision memo, a spec. Plans are simply where I reach for it most, and where it converges fastest. The looser and fuzzier the input, the more rounds it takes to agree, so non-plan runs tend to run longer - worth it when the answer matters, overkill for a quick lookup. For that, single-shot ask-gpt, ask-gemini, ask-grok, and ask-all (which fans out to all three in parallel) are right there. consensus is for when it has to be right.

Nothing about this is Terraform, or even code. That generality is why it is the headline.

The quieter two

Same release, narrower scope:

code-intelligence - a language-agnostic skill. Agents grab text grep when they should ask the language server, and silently swap tools when one is missing, then report "found all references." This encodes search precedence: the language server (LSP) for symbols, rg/ripgrep for exact text, an embedding/semantic grep (such as mgrep) for fuzzy discovery - and a hard rule to disclose any substitution on the first line of the reply. You learn the moment coverage drops.
terraform-skill - routes a Terraform/OpenTofu request to its real failure mode (identity churn, blast radius, state corruption) before emitting HCL. It is terraform-ls aware: it knows the language server has no rename provider, so it runs the safe manual reference workflow instead of a blind find-replace. It is approaching 2,000 GitHub stars - the part of this post I am quietly proud of.

Those two are discipline for the agent's hands. Consensus is discipline for its judgment, and judgment generalizes further.

A note on claude-delegator

I did not invent the delegation layer. claude-delegator is a fork of Jarrod Watts' original (MIT, upstream currently quiet), fully based on his design - I kept the structure and the license.

What I added is what months of daily use exposed: a Gemini bridge that wraps Google's Antigravity CLI (agy) with auto-gemini-3 as the routing default and recovers an answer the CLI flushed to disk after a soft timeout instead of failing the call; a fresh Grok bridge over the xAI API that is advisory-only but reads attached files via the xAI Files API (with TTL-based cleanup); two more experts (Researcher and Debugger) on top of the original five; severity-graded reviews so three parallel reports merge cleanly; and a hardened /consensus loop where Claude pre-commits a blind verdict before any external sees the artifact, with a Stage 2 blind cross-review on top (adapted from karpathy/llm-council). Plus the bundled ask-gpt / ask-gemini / ask-grok / ask-all / consensus commands, so the workflow ships with the plugin instead of living in my dotfiles. The seven expert prompts borrow from oh-my-openagent and PAL; both credited in the README.

Credit for the foundation is his; the bug-fixing scar tissue is mine.

Why a plugin, not a blog post

One honest caveat, because I have hit it myself: skills are model-triggered, which makes them soft. Packaging this as a plugin improves reuse and discoverability. It does not guarantee the agent obeys every time - hard enforcement (a real pre-tool gate) is a separate, still-open problem.

I keep finding and fixing bugs in all three. That constant repair is the only reason I trust them enough to write this - and I would rather say so than oversell the fix.

The takeaway

Stop accepting the first confident plan from one model. Make them argue, and only move when they stop. The release is at github.com/antonbabenko/agent-plugins; consensus ships in claude-delegator from the same marketplace. The cheapest review is the one that happens before you execute.

DEV Community: Anton Babenko

modules.tf now reviews Terraform module pull requests

A profile writes down what the modules allow

Four skills apply the profile

A review ends with a verdict and a list of what was not checked

mtf-facts decides what a parser can decide

Behaviour changes are verified with escalating evidence

What it will not do

Install

This is the beginning

Deliberation Dashboard: see and analyze your consensus results

You cannot fix a panel you cannot see

The analyzer tells me which model to drop

Every run remembers the config it ran with

Slow is fine, unbounded is not

Models argue about today, not their training cutoff

Smaller things that removed friction

What I would still not trust it with

Try it

Meet Deliberation: 400+ models is easy, knowing which ones earn a place is hard.

A few words first

Install it now

What actually changed

Quick one-offs: just ask

1. Fan-out with no cross-talk

2. The consensus loop, where the voting lives

3. Synthesis, when there is no verdict to give

4. Two drivers, one rulebook

5. Make the panel stay accountable

The parts I will not oversell

The takeaway

One model is a guess. Three that agree is a plan.

One model is a guess

What /consensus actually does

The quieter two

A note on claude-delegator

Why a plugin, not a blog post

The takeaway

What `/consensus` actually does