A security check that fails but doesn't block a merge isn't a security gate. It's a warning label nobody has to read.
I learned that the hard way building pipeline-gatekeeper, a small CI/CD pipeline that wires together three open-source security scanners: Trivy for container vulnerabilities, gitleaks for exposed secrets, and Trivy IaC for insecure Terraform configuration. The goal wasn't to build new scanning tools. It was to prove that wiring existing, trusted tools together correctly actually catches what it's supposed to catch, and to find out where "correctly" breaks down in practice.
The setup
Three parallel jobs run on every pull request:
- Container Vulnerability Scan (Trivy): scans the built Docker image for CRITICAL and HIGH severity CVEs in both OS packages and Python dependencies.
- Secret Detection (gitleaks): scans the full commit history for hardcoded API keys, tokens, and credentials.
- Terraform Policy Scan (Trivy IaC, using tfsec rulesets): scans Terraform configuration for misconfigurations like unencrypted storage or missing public-access restrictions.
Trivy was the natural choice for both container and IaC scanning since it runs as a single stateless binary with no daemon or database, and using one tool for two of the three gates kept the pipeline simpler than adding a separate IaC-specific scanner.
Proving it actually works, not just that it's configured
Design decisions are easy to write down and easy to get wrong in practice, so I tested this the same way I've tested every project before it: by deliberately breaking things and watching what happened.
I opened a pull request with three real violations baked in on purpose:
- A dependency (
pillow==8.2.0) with a known critical remote-code-execution CVE - A hardcoded GitHub personal access token, formatted exactly like a real one, in a config file
- An AWS EBS volume in Terraform with
encrypted = false
All three checks failed, exactly as expected.
What I found when I looked closer
This is the part that actually mattered. Two things went wrong that I hadn't planned for, and both were more useful to find than a clean pass would have been.
First: failing checks didn't actually block the merge. GitHub showed all three checks red, and right underneath, it also showed "No conflicts with base branch. Merging can be performed automatically," with an active merge button. The scanners were detecting violations correctly. Nothing was stopping anyone from merging past them anyway. Detection and enforcement turned out to be two completely separate things, and I'd only built the first one. The fix is a branch protection rule requiring these checks to pass before merge, which GitHub Actions doesn't set up for you by default just because a workflow exists.
Second: gitleaks found the secret but couldn't tell me about it where I was looking. It tried to post a summary comment directly on the pull request and failed with a permissions error, Resource not accessible by integration. The scan still worked. It still reported the leak in the job logs and summary. It just couldn't write to the PR itself, because the default GITHUB_TOKEN didn't have pull-requests: write permission. An easy fix once I saw it, but exactly the kind of thing that looks fine in a demo and quietly fails in a way you don't notice until you go looking for the comment that should be there.
Why I'm glad I found these instead of avoiding them
It would have been easy to stop at "all three checks failed as expected" and call the project done. That's what the first version of my own README said, before I looked closely enough at the actual PR to notice the merge button was still active.
The gap between "the check fails" and "the merge is blocked" is exactly the kind of thing that's invisible until someone tests it end to end, and it's a real, common misconfiguration, not a contrived edge case. A pipeline that reports failures nobody has to act on isn't protecting anything. It's just generating logs.
After fixing both issues
What this project didn't try to be
This isn't a production security platform, and three scanners wired into one workflow isn't a complete security posture. It doesn't cover runtime security, dependency license compliance, or SAST for the application code itself. What it does prove, concretely, is the difference between a pipeline that looks like it's enforcing something and one that actually is, and that difference is worth checking for directly instead of assuming.
Top comments (0)