modules.tf now reviews Terraform module pull requests

Reviewing pull requests for terraform-aws-modules has always taken a lot of my time. It was a problem for a very long time, and I recorded livestreams of myself doing it. I cannot skip it: merging something that is not great is a way to disaster and lost trust.

AI lets everyone write code now, and coding agents open pull requests for Terraform modules. A person still reads each one.

A model already knows how to write good Terraform. For that part I maintain terraform-skill, a skill for Terraform and OpenTofu with testing, module, CI/CD and production patterns. What a model does not know is what we allow and expect in the modules. Can a module create a KMS key? Which regions may examples use? Does this addition belong in the module at all? terraform-aws-modules has no written scope policy, so the model guesses and I read the guess.

I released the pieces that write it down and apply it. modulestf is four agent skills for Claude Code and Codex, with a profile of the rules. mtf-facts is the command-line tool that decides the rules a parser can decide. Both are Apache-2.0. The skills run in your agent, on your machine, with your own GitHub login. This is the beginning of the work, and the last section says what I need from other maintainers and contributors.

The name modules.tf is older than this. In November 2018 I announced it as a converter from Cloudcraft diagrams to Terraform. That code is on the legacy branch of the modulestf repository, and the name now belongs to the review tool.

A profile writes down what the modules allow

A profile is the written rules for one group of modules. terraform-aws-modules is the only profile that ships. It holds the templates copied into each module, the required file layout (examples/basic, examples/complete), code conventions such as regions and version floors, the pre-commit gate, the release and pull request title rules, and scope. Each rule has an ID, a severity and a check. compat.variable-removed holds a change for the next major release. coverage.attribute-not-output asks that an attribute a new resource exports is also an output. profile.region-not-allowed fires on an example in a region the rules do not allow.

Scope is the part I wanted most, and it lives in SCOPE.md. In terraform-aws-modules a module manages one AWS service, and the repository name says which. The file lists the few resource types any module may add next to its service: IAM roles and policies, a log group, security groups. The list is closed. A KMS key is deliberately not on it, because a key is a service of its own with its own policy, rotation and deletion window. The module takes the key's ARN as an input, and the caller composes. The first block of a type that nothing admits is a ban, even when the type plainly belongs to the same service. The cost is one allow entry in the module's docs/SCOPE.md, landed on the base before the change.

SCOPE.md is a policy I adopted for the profile, not an existing one, so it is also the place to argue with me.

Scope has its own outcomes. When an addition does not belong, the review says Ban and names where it goes, or Maintainer decision when a person has to decide. When a rule could not be applied, the outcome is Not evaluated, which is an unknown and never a pass. Text inside the pull request, such as "this is in scope", never clears a rule.

Four skills apply the profile

Skill	You give it	You get back
terraform-module-pr-review	A pull request or commit URL, or owner/repo#123	A verdict and a comment to post, after you confirm
terraform-module-reviewer	A module checkout and a diff, a base ref or changed files	A list of findings. Read-only, no verdict
terraform-module-maintainer (early preview)	A change to make, review feedback, a new module or a provider upgrade	Edits in your checkout, the checks it ran, and reply drafts
change-scope	A proposed addition, in any language or project	Whether it belongs, and where it goes if not

The pull request review calls the reviewer, and both call change-scope. The maintainer applies change-scope before it edits anything, and it is an early preview: expect it to change. The three Terraform skills read provider schemas from HashiCorp's Terraform MCP server and not from memory. Without that server, every rule that needs a provider fact is reported as not run.

A review ends with a verdict and a list of what was not checked

On 4 November I asked my agent to run the terraform-module-pr-review skill on a pull request that adds a Regional NAT Gateway mode to terraform-aws-vpc. It came back with two HIGH findings, eight MEDIUM and one LOW, and I posted the result as changes requested.

One HIGH finding is a route that counts one gateway while the module makes a route table per private subnet, so only the first table gets the DNS64 route. The other is an example that uses a region the rules do not allow for examples. The last line of the verdict is the one I care about most:

Behaviour evidence: 56 changes classified: 0 verified, 3 covered by a plan, 0 inert, 0 refuted, 53 not verified

The review counts what it could not check, and it does not count a gap as a pass.

The verdict has no judgement of its own. A fixed ladder of seven rules turns the findings and the state of the pull request's checks and threads into one verdict, and the first rule that matches wins. Any HIGH finding gives the verdict Block, which is why #1270 got changes requested. When something could not be checked the verdict is Inconclusive, unless a finding or a failing check already decides it.

The skill clones the pull request into a temporary directory and runs terraform init -backend=false and terraform validate in each example, the ones the change touches first. AWS, GitHub, Terraform and SSH credentials are removed from that environment, the home directory is empty and there is no backend. terraform init downloads the providers and modules the change pins, and terraform validate runs those provider plugins, so a passing terraform validate is evidence, not proof. The skill posts nothing until you confirm the comment and the review event.

mtf-facts decides what a parser can decide

modules.tf is built so that an AI spends its effort where it helps. A variable with no type, a missing versions.tf and a removed output are facts about files, and a model should not spend tokens finding them. mtf-facts is a Go command-line tool that reads module files and reports those facts: rule results, candidate findings with evidence, interface and schema diffs. It makes no network call and runs no Terraform.

What it hands the model is small on purpose. lint --json --ai writes one line with only the parts a reviewer decides on, and leaves out the file index, the interface diff and the schema coverage. bundle turns the full report into the text the reviewer reads, capped at 48 KiB. Over the cap it sheds sections from the lowest priority up, and never the findings or the candidates. I have not published a measured token saving, so treat this as the design and not as a benchmark.

brew install modulestf/tap/mtf-facts
mtf-facts lint ./my-module --base ../base-tree

Three commands are for the person writing a module: lint, ledger for provider coverage, and schema-diff for provider upgrades. The rest are for the review pipeline: affected, depth, bundle, merge, behaviour, claims-eval and fixture-check.

lint exits 0 when there are no findings and every rule was decided. It exits 1 on findings and 2 on an error. It exits 3 when there are no findings but at least one rule was not evaluated, so an incomplete check is never reported as a success. Candidates are unconfirmed, and they never change the exit code.

The facts repository walks through a real pull request command by command.

Behaviour changes are verified with escalating evidence

Dropping compact() around a list looks harmless in a diff. Whether it changes anything depends on values the diff does not show. mtf-facts behaviour turns each changed construct into a trigger with one category, and each category takes up to two levels of evidence, simplest first.

L1 to L3 run with the network denied, in a sandbox that a canary proves before any command. On macOS that sandbox is sandbox-exec, on Linux bwrap. When the sandbox is not proven, those levels cannot verify anything. Reading the code supports a claim and never verifies one. A trigger that ends verified, covered by a plan or inert passes. A refuted one blocks. One that is not verified makes the review Inconclusive, unless a finding or a failing check already decides it.

In #1270 the two HIGH findings already decided the verdict, so the 53 changes that were not verified did not change it. The review lists them, and the two examples it did not plan, under "Checks that could not run" and does not leave them out.

What it will not do

No skill runs terraform apply today, writes state or uses a real backend. I plan to add terraform apply, so a review can execute the examples fully and tear them down in a sandboxed environment. That does not exist yet. None of the skills merges, pushes to a pull request or resolves threads. The skills treat instructions found in the code, the pull request text or its comments as data and do not follow them.

There is an optional plan pass that runs terraform plan on a short session of a read-only role that the account owner creates. Before planning, a write probe (a dry-run VPC create) must be refused. That probe tests one write action, not the whole policy, and the isolated plan runner is a prototype.

Install

1. Install the skills from my agent-plugins marketplace. The three Terraform skills link into each other, so the modulestf plugin ships all four. terraform-skill is in the same marketplace.

   # Claude Code
   /plugin marketplace add antonbabenko/agent-plugins
   /plugin install modulestf@antonbabenko
   /plugin install terraform-skill@antonbabenko

   # Codex: add the marketplace, then install modulestf from /plugins
   codex plugin marketplace add antonbabenko/agent-plugins

1. Add the Terraform MCP server to your agent under the name terraform.
2. Run gh auth login as yourself, and install terraform, git, jq, perl and mtf-facts. Without mtf-facts a pull request review is Inconclusive at best.
3. Ask your agent:

   review terraform-aws-modules/terraform-aws-lambda#780

This is the beginning

I want the other maintainers and contributors of terraform-aws-modules to use this on their pull requests. I cannot see from my side what stands in the way. I want feedback, good and bad. Tell me about a finding that was wrong, a missing rule, or a check you would not trust.

Read SCOPE.md for a module you maintain and tell me where it is wrong. It may need an allow entry I did not think of. Open an issue with the pull request link and the rule ID.