Day 25/90: It passed validate AND plan, then shipped a broken security group

#terraform #go #devsecops #testing

Day 25/90: Terraform testing with Terratest

90 Day Security Infrastructure Challenge

A teammate's module change passed terraform validate, passed terraform plan with no errors, collected two approvals, and shipped a security group that no longer allowed 443. The module's whole job was to open HTTPS to the public load balancer. A refactor the week before had moved ingress into a dynamic block fed by a variable, and 443 fell out of that variable's list. Validate and plan both stayed green, because neither one runs the apply or reads the live resource. The first thing that noticed was a failing health check.

This post is the case for behavioral testing of IaC, and the structure of a Terratest test that would have caught it.

What validate and plan actually prove

terraform validate proves your configuration is well-formed: types line up, references resolve, blocks are legal. terraform plan proves what Terraform intends to change by diffing your config against state. Both are useful and both are fast. But notice the boundary: neither runs apply, neither calls the live cloud API, and neither asks the deployed security group whether it really allows 443. A module can be internally consistent and externally wrong, and that is precisely the failure these two tools are blind to.

Consider the module's output -- the surface a test would read:

output "sg_id" {
  value = aws_security_group.web.id
}

output "open_ports" {
  description = "Ports the web SG accepts on tcp"
  value       = sort([for rule in aws_security_group.web.ingress : rule.from_port])
}

Plan will happily show this output being computed. It cannot tell you the value is wrong, because the value depends on a variable that lost 443 during a refactor. Only an apply produces the real list, and only an assertion against that real list catches the regression.

A Terratest test is a Go test that drives the CLI

Terratest is a Go library from Gruntwork. A test is an ordinary go test function. It builds a terraform.Options, runs InitAndApply against a sandbox account, reads outputs, asserts with testify, and destroys in a defer. Here is one that pins the contract "this SG accepts 443":

package test

import (
    "testing"

    "github.com/gruntwork-io/terratest/modules/random"
    "github.com/gruntwork-io/terratest/modules/terraform"
    "github.com/stretchr/testify/require"
)

func TestWebSGOpenPorts(t *testing.T) {
    t.Parallel()

    opts := &terraform.Options{
        TerraformDir: "../modules/web-sg",
        Vars: map[string]interface{}{
            "name_suffix": random.UniqueId(),
            "vpc_id":      "vpc-0abc123",
        },
        NoColor: true,
    }

    defer terraform.Destroy(t, opts)
    terraform.InitAndApply(t, opts)

    openPorts := terraform.OutputList(t, opts, "open_ports")
    require.Contains(t, openPorts, "443", "web SG must accept HTTPS")
    require.Contains(t, openPorts, "80", "web SG must accept HTTP for redirect")
}

require.Contains (vs assert.Contains) stops the test on the first failed expectation, which is what you want when a missing 443 makes every later check meaningless. t.Parallel() plus the random.UniqueId() suffix lets several module tests run concurrently without colliding on resource names.

Why the static layers could not catch this

The instinct is "we should have had a policy scan." But run the broken module through checkov or tfsec and they pass. A security group with no 443 rule is valid and secure-looking. Static scanners flag dangerous presence (a public port 22, an unencrypted volume), not intended absence. The missing rule is invisible to every tool that only reads code:

# the cheap layers -- all green on the broken module
terraform fmt -check -recursive
terraform validate
tflint --recursive
checkov -d . --quiet      # finds 0.0.0.0/0 on 22, NOT a missing 443

That is the gap a behavioral test fills. The static tools answer "is this code well-formed and free of known bad patterns?" Terratest answers "does the thing this code builds actually do what we claimed?"

The trade-off: gate it, do not spam it

Terratest creates real resources, so it is slow (minutes) and costs money each run. The discipline is a pyramid:

Base, every save, free: fmt, validate, tflint.
Middle, every PR, cheap: checkov / tfsec on the plan.
Top, gated in CI, slow + costs money: Terratest against a sandbox.

Run Terratest on a merge gate or a nightly job, never on file save. The cheap layers carry the constant load; the expensive layer proves the handful of claims that only a real apply can prove. The broken-443 regression lives at the top of that pyramid -- and that is exactly why the pyramid has a top.

Why this belongs in a security infrastructure track

A dropped 443 rule is an availability incident. A dropped deny rule, a widened CIDR, or an encryption flag silently flipping to false during a refactor are security incidents -- and they pass validate and plan the same way. Behavioral testing is how you keep a refactor from quietly changing what your infrastructure permits. Static checks describe intent. Terratest proves behavior.

DEV Community