DEV Community: Victor Okoroafor

My Policy-as-Code Check Passed. It Never Read the Policy.

Victor Okoroafor — Mon, 19 Oct 2026 15:27:22 +0000

In my last post, I showed how OPA and Conftest block over-privileged IAM roles before anything gets deployed. I ended with a warning: there's a case where the check passes when it shouldn't.

This is that case. It's worse than a missed wildcard, because nothing tells you it happened.

The setup

My deny rule starts like this:

policy := json.unmarshal(resource.change.after.policy)
statement := policy.Statement[_]
has_wildcard(statement.Action)

It reads the policy from the Terraform plan, parses it, and looks for a wildcard. It works perfectly when Terraform already knows the full policy at plan time.

Real policies are rarely that simple.

The policy Terraform can't see yet

Most IAM policies point at resources you're creating in the same apply. To test this, I wrote a small Terraform config with a DynamoDB table, a Lambda role, and this policy:

resource "aws_iam_role_policy" "lambda_dynamodb" {
  role = aws_iam_role.lambda.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = "*"
      Resource = aws_dynamodb_table.transactions.arn
    }]
  })
}

On a fresh deploy, the table doesn't exist, so its ARN doesn't exist either. Terraform can't build the final JSON string, so it marks the whole policy as unknown:

In the plan JSON, that looks like this:

"after": { "name": "lambda_dynamodb" },
"after_unknown": { "policy": true }

What OPA does with nothing

My rule reads resource.change.after.policy, and that field isn't there. In Rego, a missing field isn't an error. It's undefined, and an undefined line means the rule body doesn't match.

So the deny rule doesn't fire. No deny means no violation. No violation means Conftest exits 0.

Here's the real run. The wildcard is right there on line 55 of main.tf, Terraform confirms the policy is unknown, and my rules pass it anyway:

Honestly, my first reaction was confusion. The wildcard was right there in the code, and the check was green. It took me a while to realise the rule hadn't approved the policy. It had never seen it.

The check didn't say yes. It said nothing, and my pipeline would have treated silence as a yes. In my pipeline, the apply job only runs after a green Conftest step.

Why this is the dangerous one

A missed edge case like "s3:*" is a gap you can see in the rule. My own rule has that gap too. But this one is worse, because it hides in the input. The rule is correct. The data it needs just isn't there, and a deny-only policy fails open by design: it only blocks what it can prove is bad.

In security terms, that's backwards. If a gate can't evaluate something, it should stop it, not wave it through.

The fix: fail closed

I added a rule that denies any IAM policy Terraform can't resolve at plan time:

deny contains msg if {
    resource := input.resource_changes[_]
    iam_policy_resources[resource.type]
    not is_excluded_policy(resource.name)
    resource.change.after_unknown.policy == true

    msg := sprintf("IAM policy '%s' can't be evaluated at plan time (known after apply): failing closed", [resource.name])
}

iam_policy_resources and is_excluded_policy are helpers already in my policy file. The full file is in the repo.

Same plan, new rule:

Now an unreadable policy is a failed check, not a silent pass. In CI, that failure stops the pipeline before apply.

Then I turned it on in my own project

This is where it got real. I ran the new rule against the Terraform for my final-year project, and it failed on two of my own IAM policies. Both pointed at resources created in the same apply, a DynamoDB table and a Cognito user pool. My old rules had been passing them without reading them.

So the fail-closed rule did its job. It also meant my deploys would fail until I fixed the policies. There were two ways to deal with that:

Split the apply. Create the table first, then the policy on the next run, when the ARN is known.
Build the ARN from values Terraform already knows, like the account ID, region, and a fixed table name, so the policy resolves at plan time.

I went with the second. It keeps a single apply, and because the policy is readable at plan time, my wildcard rule can check it again. The fail-closed rule stays as a safety net for anything that still can't be resolved.

data "aws_caller_identity" "current" {}
data "aws_region" "current" {}

resource "aws_iam_role_policy" "lambda_dynamodb" {
  role = aws_iam_role.lambda_execution.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:UpdateItem", "dynamodb:Query"]
      Resource = "arn:aws:dynamodb:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:table/${var.transactions_table_name}"
    }]
  })
}

The table and the policy both get their name from one shared local, so they can't drift apart.

Where this trick stops working

Building ARNs only works for names you control. Cognito was the exception: AWS generates the user pool ID when it creates the pool, so there's no way to write the exact ARN at plan time.

For that one, I scoped the policy to user pools in my own account and region, with read-only actions:

Resource = "arn:aws:cognito-idp:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:userpool/*"

It's broader than one pool, but much narrower than "*". I chose that over adding an exception to the fail-closed rule, because exceptions are how silent passes come back.

After the fix, no IAM policy in the plan is unknown, and the pipeline goes green for the right reason this time:

The lesson

Every policy-as-code rule makes an assumption about its input. Mine assumed the policy would be there.

When you write a deny rule, ask one more question: what happens when the field I'm reading is missing? If the answer is "the rule doesn't fire," you've built a gate that fails open.

Test your gates with bad data. Then test them with no data.

Full code, including the fail-closed rule: github.com/vicGrey/aws-serverless-zero-trust-governance

I Built an Offline AWS Auditor. It Caught My Mistakes, Then Made Its Own.

Victor Okoroafor — Sun, 18 Oct 2026 22:11:09 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

My first user (it was me)

My final-year project was a serverless fintech backend on AWS: 48 Terraform resources and 10 IAM policies, all written by me, a cybersecurity student who thought he knew what he was doing.

I was the target user. A small team, no security engineer, permissions written under deadline. So before offering this to anyone else, I ran ZeroLeak on my own plan with the Wi-Fi off. It found things I'd missed, and it handed me two "fixes" that would have broken my own app. More on that below.

Would I pay for it? Not for the tool, which is free and open source. I'd pay for the reviewed report, because I know exactly how confidently wrong my own tool can be. I only know that because I was the customer first.

The people I actually want to help look a lot like the person who wrote that project. Meet Ada. She's the CTO of a six-person fintech startup. (She's made up, but if you've worked at a startup, you've met her.) Her team ships on AWS fast, and nobody's job is security. She knows their permissions are messy, and she can't say which role is the dangerous one.

She has three options: hire a security firm she can't afford, paste her AWS setup into a cloud chatbot and hope nobody minds, or ignore it until something breaks.

I just graduated in cybersecurity, and I want to be her fourth option. This post is about the tool I built to make that offer, and about the three times it told me everything was fine when it wasn't.

What I built

Picture an AWS account as an office building. IAM policies are the keycards: they decide who can open which doors. Most cloud breaches don't involve anyone breaking a door down. Someone simply holds a keycard that opens far more than it should. In a policy, a * is a master key.

ZeroLeak AI reviews those keycards entirely on a laptop. No cloud AI, no API keys, and no internet once it's set up. You give it one policy, or a whole Terraform plan (the blueprint a team uses to build their AWS setup), and two very different reviewers look at it:

A security guard with a checklist. Conftest runs Rego rules (OPA) and checks for known-bad patterns in about 20 milliseconds. It's strict, fast, and impossible to sweet-talk.
A detective. Gemma 2 2B, an open AI model running locally through Ollama, explains the risks in plain English and suggests a tighter policy.

The guard then re-checks the detective's suggestion. The report lists exactly which permissions the fix removed, and everything exports to a PDF Ada can hand to her team or her investors.

Test 04 after codifying Rule 4: the guard now fails it, and Gemma explains why.

The side hustle

Here's what I'd offer Ada:

She sends me her IAM policy documents under an NDA. Not her whole Terraform plan, since plans can contain secrets.
I run them through ZeroLeak on my own laptop, check every finding by hand, and send back a PDF report with proposed fixes.
No AI company ever sees her infrastructure.

My pilot price is $100 for up to 10 policies, with the first few reviews discounted in exchange for honest feedback. I don't have a paying client yet. This post is part of finding the first one.

Demo

The first docker compose up needs internet once to download the app and the model. After that, the whole audit, PDF included, runs with the network off. Telemetry is disabled (gatherUsageStats = false). You can test the privacy claim yourself by switching off your Wi-Fi.

Page one of a Test 04 report. On this run the gate refused Gemma's fix, and the report says so up front (more on why below).

Code

vicGrey / zeroleak

🛡️ ZeroLeak AI

Offline AWS IAM Policy Auditor

A deterministic rule gate (OPA / Conftest Rego) plus a local open-weight model (Google Gemma 2 2B)

🔒 No cloud endpoints. No API tokens. No vendor DPA. $0 cloud spend.

~20 ms
rule gate per policy

52.2 s
mean Gemma 2 2B on CPU

0 bytes
leave the machine

10
hand-crafted benchmark policies

$0
cloud spend

Architecture • Discovery Loop • Benchmark • Real Plan • Quickstart • Wi-Fi Off Test • Limits

💡 The problem: cloud LLMs send your infrastructure secrets off-machine

IAM policies are a map of an organization's infrastructure: account IDs, role naming conventions, ARN hierarchies, bucket names. Auditing them with a cloud LLM API means sending that map to a third party.

For freelance cloud security consultants and DevSecOps contractors working under NDAs, and for regulated teams (HIPAA, PCI-DSS, SOC 2), that's often not allowed without a signed…

View on GitHub

git clone https://github.com/vicGrey/zeroleak.git
cd zeroleak
docker compose up   # then open http://localhost:8501

MIT licensed. The baseline wildcard rules are adapted from my final-year project, aws-serverless-zero-trust-governance. Everything else (the Gemma pipeline, the combination rule, the Terraform plan parser, and the reports) was built during the challenge week.

How I built it

Gemma 2 2B runs locally in Ollama at temperature 0.1 (low randomness, so it gives near-identical answers on every run). Each policy gets one structured prompt that returns both the explanation and the fixed JSON, and a sanitizer repairs the JSON when the model cuts off on a long policy. Conftest/OPA runs the Rego v1 rules, Streamlit is the interface, WeasyPrint makes the PDFs, and Docker Compose runs the app next to an Ollama container that downloads gemma2:2b on first start. A small parser pulls every IAM policy out of terraform show -json.

I picked the 2B model on purpose. It's a ~1.6 GB download that runs in 3 to 4 GB of RAM on an ordinary laptop, and that's the kind of machine a freelancer actually owns. The cost is speed: 52.2 seconds per policy on average (39.0s to 67.3s) on a CPU. Newer and bigger Gemma releases exist. The model is one tag in Ollama, so you can swap it, but I haven't tested a larger one here.

I tested it on 10 policies I wrote, each targeting a known IAM mistake:

#	What the policy does wrong	Guard (rules)	Detective (Gemma)
01	Nothing (clean control)	0, correct	2, false positive
02	Wildcard action	1	1
03	Wildcard resource	1	1
04	PassRole + Lambda escalation	0, missed	1, caught
05	Over-broad DynamoDB access	2	1
06	Same risk, different JSON format	2	2
07	`NotAction` loophole	2	1
08	Wildcard `sts:AssumeRole`	1	1
09	EC2 instance-profile escalation	1 (wildcard only)	2 (incl. escalation)
10	Prompt injection hidden in a label	2	resisted

The guard caught almost everything, in milliseconds. The detective earned its 52 seconds on the two rows where the danger was a combination of permissions the guard had no rule for (and gave me my first false positive on row 01, flagging standard logging as a hazard).

Getting to a table I could trust was the hard part.

Three green checkmarks I couldn't trust

1. The guard wasn't looking. Test 04 lets someone create a Lambda function and hand it a more powerful role than their own. My first pipeline said: original policy, 0 violations; Gemma flags the danger and writes a fix; fixed policy, 0 violations. Two green checkmarks, "Remediated and Verified."

But the guard's checklist only covered master keys. It had no line for this combination, so it waved through the dangerous original and the fix. A broken fix would have passed just the same. Emma, a friend who freelances on Upwork, pointed out that the verification was circular.

The fix was a loop: when the detective finds something new, a human writes it into the guard's checklist.

import rego.v1

deny contains msg if {
    some s1 in get_statements
    s1.Effect == "Allow"
    has_action(s1, "iam:PassRole")
    some s2 in get_statements
    s2.Effect == "Allow"
    has_action(s2, "lambda:CreateFunction")
    msg := "[passrole_lambda_escalation] PassRole with lambda:CreateFunction allows privilege escalation."
}

# helpers in policies/iam_policy.rego

Now the original fails (1 violation), and in my logged run the fix passed (0). The green check finally means something.

It also works against me. On a later run, Gemma kept both actions but scoped them to a single function and a single role, role/role-name, a placeholder ZeroLeak flags before deployment. The gate still refused it (RULE GATE: FAILED, 1 remaining), because my rule checks which actions are present, not which resources they touch. Whether that fix is safe depends on what role-name can do. That's a call for a human, not for the rule or the model.

Test 09 has the same gap until someone writes its rule too: nothing in my ruleset tests for PassRole + RunInstances, so any "verified" there only means "passed the wildcard rules."

2. My own real project. I ran ZeroLeak on that same plan, with the Wi-Fi off. It took 102.29 seconds. Of the 10 policies, 5 passed, 2 failed, and 3 couldn't be read at all.

Those 3 pointed at things that don't exist until deployment (a bucket, a database table, a login pool), so they show up as "unknown" in a plan. A scanner that treats unknown as "nothing to flag" calls them clean. ZeroLeak now lists them as NOT EVALUATED (KNOWN AFTER APPLY), so the client gets a to-do list instead of false comfort.

My final-year project's plan: 48 resources, 10 policies, 3 not evaluated at plan time. Runtime varies by machine and model state; this rerun took 88s, while the logged run took 102.29s.

The 2 failures were worse. Gemma rewrote both, the guard passed both rewrites, and both were wrong. One removed logs:PutLogEvents from a logging role, the one permission it existed for. That's like fixing a cashier's keycard by taking away access to the till. The other scoped config:PutEvaluations to a fake example ARN, but that action doesn't support resource-level permissions, so * was actually correct. The guard passed both because it only looked for a literal *, and any other text satisfied it.

The report badge used to say "RE-VERIFICATION: PASSED". It now says "RULE GATE: PASSED (Workload & ARN validity not checked)", which is the honest version.

3. The final pressure test. Three fresh policies, none used in development.

First, Gemma flagged a policy combining iam:PassRole, iam:CreateAccessKey and iam:PutUserPolicy. That one is more of a sanity check than a discovery: either of the last two is enough to escalate privileges on its own, and standard scanners like Cloudsplaining already flag them.

Second, arn:aws:s3:::*/* (every file in every bucket) slipped past the baseline guard because it isn't a bare *, so the rules now catch wildcard ARNs too.

Third, on a perfectly fine read-only policy, Gemma confidently added a Condition: "s3:BucketOwner": "arn:aws:iam::123456789012:user/my-user". In AWS S3, a valid key is s3:ResourceAccount (which takes a 12-digit account ID, not an IAM user ARN), not s3:BucketOwner. An IAM condition on a key AWS doesn't define never matches, so the Allow would silently fail and reads would be denied. The guard passed it because it only looked for wildcards.

So ZeroLeak now shows "preserved intact" only when the guard and Gemma agree. When they don't, the report says Manual Review Recommended and labels any fix as advisory (no copy-paste fix without manual verification).

So, why would Ada pay a person? Because the tool can be confidently wrong in ways that look right. ZeroLeak does the private, tireless first pass, and a human who knows what each permission is for does the part that matters.

Why does open innovation matter?

The prompt for this challenge asks why open innovation matters for what I built. Here's the specific answer, not the general one.

For Ada, three reasons:

Privacy she can verify. "Your setup never leaves my laptop" is a promise a hosted AI can't keep, however good its privacy policy. With Gemma running locally, you can check it by unplugging the internet.
A price that works. Every audit costs nothing to run, with no per-request bill, so my only real cost is my own time.
Results anyone can check. Anyone can download the same open model, run my ten benchmark policies and three pressure tests from the repo, and see if they get my results, with no account and no API key.

A big hosted model would probably be faster and smarter than a 2B model on a laptop. I'm trading that for privacy and cost, and for a rule gate that doesn't depend on the model being right.

What I'd tell Ada before she pays me

A pass means "passed my rules," not "safe." Zero violations is zero violations of the rules I wrote. My PassRole rule is deliberately broad and would also flag a properly scoped deployer role. A production rule should check Resource and require an iam:PassedToService condition.
The detective cries wolf. Small open-weight models have variance. In my benchmark run, Gemma flagged standard CloudWatch logging combined with S3 reads as two phantom combination hazards on my clean control policy (row 01). That is my first measured false positive, and it's why every flag gets human review.
It reads one policy at a time. Real attacks often chain several policies or roles together. A graph tool like PMapper maps those paths, and ZeroLeak complements it.
Fast mode skips passing policies. To keep big plans quick, Gemma only reviews policies that fail the guard, so a hidden combination like Test 04 needs single-policy mode or a rule.
The evidence is small. Ten test policies, one real plan, and three pressure tests show promise, not a proven detection rate.

I started this project thinking the AI was the product. I'm finishing it knowing the product is the judgment: knowing when a green checkmark was actually looking for the problem.

Prize categories

Best use of Gemma: Gemma 2 2B runs fully locally through Ollama as the detective in the pipeline. It flagged two risky permission combinations (tests 04 and 09) that the rule gate had no rule for. Every audit runs behind a deterministic rule gate, and because the model runs locally, no client data leaves the machine.

How to Block Over-Privileged IAM Roles in AWS Using Policy-as-Code

Victor Okoroafor — Fri, 09 Oct 2026 12:29:55 +0000

Chances are, your AWS account has an over-privileged IAM role right now, and you may not know which one.

When you build infrastructure, you write Terraform that declares your resources and assigns IAM roles to them. IAM controls who can do what in your account, so the principle of least privilege has to apply: an entity should get only the permissions it needs to do its job.

That rule gets broken constantly, and the usual shortcut is a wildcard:

{
  "Effect": "Allow",
  "Action": "*",
  "Resource": "*"
}

"Action": "*" grants a function every permission it needs, plus every permission it doesn't. If that function is ever compromised, the attacker inherits everything that role can reach. One line in a policy file becomes your blast radius.

Where the check goes

You can run terraform apply from your laptop, but most teams run it through a CI/CD pipeline. A pipeline is a list of commands, and each one finishes with an exit code. Exit code 0 means the step passed and the pipeline continues. Any other exit code means the step failed and the pipeline stops.

If you can make a security check fail with a non-zero exit code, you can halt the deployment. That's the whole idea.

The policy

To block over-privileged roles, you write a policy in Rego. OPA, the policy engine, evaluates your Rego against an input document, which here is your Terraform plan converted to JSON. If your infrastructure code contains a wildcard where it shouldn't, OPA flags it.

Here's the rule that does the work, written in OPA v1 syntax:

package main
import rego.v1

iam_policy_resources := {"aws_iam_role_policy", "aws_iam_policy"}

# Does this field contain a wildcard? Handles a single string or a list.
has_wildcard(field) if {
    is_string(field)
    field == "*"
}

has_wildcard(field) if {
    is_array(field)
    field[_] == "*"
}

deny contains msg if {
    resource := input.resource_changes[_]
    iam_policy_resources[resource.type]

    policy := json.unmarshal(resource.change.after.policy)
    statement := policy.Statement[_]
    has_wildcard(statement.Action)

    msg := sprintf("IAM policy '%s' contains wildcard (*) action: violates least privilege", [resource.name])
}

A second rule does the same check for Resource.

The part that bit me

My first version of has_wildcard only had the string check:

# my first version: only catches Action = "*"
has_wildcard(field) if {
    field == "*"
}

It caught "Action": "*". It missed "Action": ["s3:GetObject", "*"], which grants the same thing written differently. My policy let it through.

The fix is the second has_wildcard definition above. That field[_] walks every element in the array instead of treating the array as a single value, so a wildcard sitting anywhere inside the list gets caught.

That second definition is the difference between a policy that works and a policy that looks like it works.

Turning a decision into a failed pipeline

OPA only produces a decision. It doesn't decide whether your pipeline lives or dies.

That's where Conftest comes in. Conftest runs on top of OPA and prints pass or fail the way a normal software test does.

When Conftest finds a deny violation, it exits with code 1, and the GitHub runner stops. Here are the steps that matter in my workflow:

- name: Terraform Plan
  working-directory: ./terraform
  run: terraform plan -out=tfplan

- name: Convert Plan to JSON
  working-directory: ./terraform
  run: terraform show -json tfplan > tfplan.json

- name: OPA Policy Validation (Conftest)
  working-directory: ./terraform
  run: conftest test tfplan.json --policy ../policies/ --all-namespaces

The deploy job is a separate job that waits for this one:

terraform-apply:
  needs: terraform-plan-and-validate

If the validation job fails, the apply job never runs. terraform plan reads from AWS to work out what would change, but nothing gets created or modified.

Conftest failed the build. Exit code 1. Nothing was deployed.

GitHub Actions emails you when the job fails by default, and you can wire it to Slack with a webhook if that's where your team lives.

Results

I tested the policy against a deliberately over-privileged IAM role, one with "Action": "*" in its policy document. The build failed, and the deployment never ran.

The validation step added about 7.3 seconds to the pipeline. That's the cost of catching a wildcard before it reaches your account, and against a pipeline that already ran for over a minute, it was noise.

What this doesn't catch

The rule matches a bare "*" only. A service-level wildcard like "s3:*" still gets through, and catching it would need a pattern match rather than an exact string check.

It also only guards what goes through the pipeline. A change made directly in the console bypasses it, which is why my project also uses AWS Config to catch configuration drift after deployment.

My repo also excludes two platform policies, because AWS requires a wildcard resource for them. Real pipelines usually need a small exception list like that.

There's a bigger failure mode I found later, one where the check passes when it shouldn't because it can't read the policy at all. I wrote about it here: My Policy-as-Code Check Passed. It Never Read the Policy.

I'd rather tell you where the gaps are than pretend the policy is complete. A rule you trust too much is worse than no rule at all.

The point

A bare wildcard in a policy that Terraform can fully resolve at plan time never reaches AWS through this pipeline. The pipeline decides, and the pipeline says no.

That's the common case. It isn't every case, and the exception gets its own post.

The full implementation, with the Terraform, the Rego policies, and the GitHub Actions workflow, is here:

github.com/vicGrey/aws-serverless-zero-trust-governance