🌟 The CI Gate Rejected the Terraform Change—but the LLM Still Ran

How I added a deterministic trace contract to an AI Terraform reviewer so rejected infrastructure changes stop before the model is invoked.

I recently built an AI-powered Terraform review agent that combines Terrascan, GitHub Actions, AWS Lambda and Gemini.

The workflow is straightforward:

1. A pull request changes Terraform code.
2. GitHub Actions runs Terrascan.
3. The scan report is sent to an AWS Lambda function.
4. Gemini reviews the findings.
5. The pipeline accepts or rejects the change.

The project worked. A risky change produced REJECT, and GitHub Actions failed the pull request as expected.

But while reviewing the execution path, I noticed an important problem: the correct final verdict did not prove that the control was enforced at the correct boundary.

The pipeline rejected the change—but the LLM had already run.

That distinction matters in DevOps. A policy that says “do not continue” should stop the next action. It should not merely ask the next action to agree that the request should have been stopped.

This article shows how I used AgentInspect to make that path visible and add a deterministic CI contract around it.

Disclosure: I tested AgentInspect independently in the workflow described here. The maintainer reviewed the AgentInspect commands for technical accuracy; the conclusions are my own. AgentInspect did not replace Terrascan, GitHub Actions, AWS controls or the application’s security policy.

The original review architecture

My Terraform review project uses a practical serverless workflow:

Terraform pull request
        ↓
GitHub Actions
        ↓
Terrascan JSON report
        ↓
AWS Lambda
        ↓
Gemini review
        ↓
APPROVE | APPROVE_WITH_CHANGES | REJECT

The Lambda function extracts relevant Terrascan findings and sends a bounded structure to Gemini. The prompt contains explicit decision rules:

• Reject when a HIGH or CRITICAL issue exists.
• Reject when there are four or more MEDIUM issues.
• Reject when the Application Load Balancer has no HTTPS listener.
• Approve with changes for one to three MEDIUM issues.
• Approve when only LOW or INFO issues remain.

The GitHub Actions workflow then reads the returned verdict and exits with status 1 when the model returns REJECT.

At first, this looked like a security gate. The result was correct and the PR was blocked.

The problem was where the authority lived.

In the original Lambda implementation, the sequence was effectively:

findings = extract_relevant_findings(results)
prompt = build_prompt(findings)

ai_review = call_gemini(prompt)
verdict = extract_verdict(ai_review)

The risk thresholds were described inside the prompt. They were not evaluated as a deterministic control before the provider call.

This creates three different concerns:

1. An obvious rejection still consumes a model call.
2. Prompt behavior can change even when the policy has not changed.
3. A correct final verdict can hide an incorrect execution path.

The third concern is the easiest one to miss in ordinary CI output.

---

A passing test was not enough

Imagine a fixture containing a HIGH-severity public-access violation. A conventional test might assert:

expect(result.verdict).toBe("REJECT");

That assertion passes whether the pipeline rejects before Gemini or calls Gemini and then accepts its rejection.

Those paths are not equivalent:

Desired
Terrascan → deterministic policy → REJECT → stop

Original
Terrascan → Gemini → parse response → REJECT → stop

The final value is the same. The control boundary is different.

For the blocked test case, I wanted to assert a stronger invariant:

Terrascan and the deterministic policy check must execute, and the LLM path must execute zero times.

That is an execution contract, not an answer-quality evaluation.

---

Adding a small AgentInspect wrapper

The existing Lambda is written in Python, while AgentInspect is a TypeScript-first toolkit. I did not pretend that it could automatically instrument the Python function.

Instead, I added a small Node.js evidence runner at the CI boundary. It wraps the operations the pipeline owns: reading the scan, evaluating the policy and, only when appropriate, invoking the existing Lambda.

For this test, I used AgentInspect 6.17.4 and pinned the version so the CI behavior would not move underneath the experiment:

npm install agent-inspect@6.17.4

The simplified runner looks like this:

import { inspectRun, step } from "agent-inspect";

const traceDir = ".agent-inspect/terraform-review";

function evaluatePolicy(report) {
  const violations = report.violations ?? [];
  const severities = violations.map((item) =>
    String(item.severity ?? "").toUpperCase()
  );

  const highOrCritical = severities.some(
    (severity) => severity === "HIGH" || severity === "CRITICAL"
  );
  const mediumCount = severities.filter(
    (severity) => severity === "MEDIUM"
  ).length;

  if (highOrCritical || mediumCount >= 4) {
    return { verdict: "REJECT", reason: "risk-threshold" };
  }

  if (mediumCount >= 1) {
    return {
      verdict: "APPROVE_WITH_CHANGES",
      reason: "medium-findings"
    };
  }

  return { verdict: "APPROVE", reason: "low-or-info-only" };
}

const result = await inspectRun(
  "terraform-ai-review",
  async () => {
    const report = await step.tool("terrascan", () =>
      readTerrascanReport("terrascan_report.json")
    );

    const policy = await step.tool("evaluate_policy", () =>
      evaluatePolicy(report)
    );

    if (policy.verdict === "REJECT") {
      return policy;
    }

    const review = await step.llm("gemini-2.5-flash", () =>
      invokeTerraformReviewLambda(report)
    );

    return {
      ...review,
      verdict: policy.verdict
    };
  },
  {
    traceDir,
    silent: process.env.CI === "true",
    metadata: {
      workflow: "terraform-ai-review",
      fixture: "high-severity-public-access"
    }
  }
);

console.log(result);

This example deliberately keeps the policy small. The HTTPS rule needs its own structured input or a stable mapping from specific Terrascan findings. I would not implement it by searching arbitrary free text and call that deterministic.

The important change is architectural: code owns the risk threshold and final CI authority; the model can provide explanation and remediation only after the deterministic gate allows that path.

---

The trace exposed the difference

I ran a controlled fixture representing a HIGH-severity finding and inspected the local trace:

npx agent-inspect list --dir .agent-inspect/terraform-review
npx agent-inspect view <run-id> --dir .agent-inspect/terraform-review

Before the short-circuit, the execution contained an LLM step:

terraform-ai-review
├─ tool:terrascan              success
└─ llm:gemini-2.5-flash       success

After moving policy enforcement ahead of the provider call, the rejected path became:

terraform-ai-review
├─ tool:terrascan              success
└─ tool:evaluate_policy        success

The absence of the LLM step was now visible, but I did not want reviewers to verify it manually on every pull request. The next step was turning it into a deterministic check.

---

Defining the blocked-path contract

AgentInspect supports deterministic checks over retained local traces. For this case, I used a JSON check configuration:

{
  "checks": {
    "tool": {
      "required": ["terrascan", "evaluate_policy"]
    },
    "llm": {
      "maxCalls": 0
    }
  }
}

Then CI checks the trace produced by the blocked fixture:

npx agent-inspect check .agent-inspect/terraform-review \
  --config blocked-path.check.json \
  --require-completed \
  --json

The broken path fails because the model call count is greater than zero. The fixed path passes because:

• The run completed.
• terrascan appeared.
• evaluate_policy appeared.
• No LLM step appeared.

This is the exact behavior I wanted from CI. It does not ask another model whether the trace looks safe. It evaluates a small, reproducible structural contract and returns a deterministic exit code.

---

Keeping evidence with the failed build

A CI failure without usable evidence usually starts another debugging cycle. To retain a bounded report, I added an artifact step:

- name: Run blocked-path regression
  run: node scripts/run-blocked-policy-case.mjs

- name: Check blocked-path contract
  run: |
    npx agent-inspect check .agent-inspect/terraform-review \
      --config blocked-path.check.json \
      --require-completed \
      --json > trace-contract-result.json

- name: Build safe trace artifacts
  if: always()
  run: |
    npx agent-inspect artifacts .agent-inspect/terraform-review \
      --output-dir ./agent-inspect-artifacts \
      --github-summary "$GITHUB_STEP_SUMMARY"

- name: Upload review evidence
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: terraform-agent-trace
    path: |
      agent-inspect-artifacts/
      trace-contract-result.json
    retention-days: 14

AgentInspect creates the local report; GitHub Actions owns the upload and retention. For real pipeline data, I would still review the exact derived artifact before sharing it outside the repository. Redaction and safety scans are safeguards, not compliance certification.

What changed in my mental model

My first implementation treated the final verdict as proof that the gate worked. It was only proof that the workflow ended with the expected string.

The stronger DevOps questions are:

• Was the declared control actually executed?
• Did it execute before the model or side effect?
• Did a rejected path stop immediately?
• Can CI prove those facts without another probabilistic judgment?

This is where execution traces are useful. They provide evidence about the path rather than only the answer.

There is also a useful separation of responsibilities:

Layer	Responsibility
Terrascan	Detect infrastructure findings
Deterministic policy code	Enforce explicit risk thresholds
Gemini	Explain findings and suggest remediation on allowed paths
AgentInspect	Record and check the execution path
GitHub Actions	Enforce the build result and retain reviewed artifacts

AgentInspect did not make the Terraform deployment secure. It helped me verify whether my own control flow matched the policy I claimed to enforce.

---

What this check does not replace

A green trace contract is narrow evidence. It does not prove that every Terraform rule is correct or that the deployed infrastructure is safe.

This workflow still needs:

• Terrascan or another infrastructure scanner.
• IAM least privilege and protected deployment credentials.
• Branch protection and human review.
• Tests for malformed and incomplete scan reports.
• Provider timeouts, budgets and rate limits.
• Prompt-injection and adversarial testing for the explanatory model path.
• Production monitoring and incident controls.

It also does not prove that Gemini’s remediation advice is good. That requires separate evaluation.

The contract proves one specific invariant: when deterministic policy rejects a Terraform change, the model path does not run.

---

Final takeaway

AI can add useful context to DevOps workflows, but it should not own controls that can be expressed clearly in code.

My Terraform review agent already produced the expected rejection. AgentInspect showed me that the route to that answer was weaker than the answer itself suggested.

Moving the risk threshold ahead of the LLM call gave the workflow a cleaner authority boundary. Adding a trace contract made that boundary reviewable in CI.

The lesson is simple:

Do not test only what your agent returned. Test which actions it was allowed to take before returning it.

---

Project links

• AI DevOps Agent repository
• AI Terraform Review Agent
• AgentInspect
• Trace contracts
• CI artifact workflow