Continuing from the part of my series that explains the program that I helped create with a team of developers, here I pressed forward with my discovery into software engineering and cybersecurity, where I wired three security scans; dependency (SCA), static analysis (SAST), and secrets, into GitHub Actions for a codebase I'd already hand-scanned once. All three came back clean, and that should have been the end of it. Instead, I made myself answer a harder question: does "no leaks found" actually mean no leaks exist, or does it just mean the scanner didn't find any the way it's currently configured? We ought to be aware that these are not the same claim, and treating them as interchangeable is how a real gap survives behind a passing pipeline.
Testing the scanner instead of trusting it
I ran two constructed AWS-key test cases through gitleaks to see whether its detection actually behaved the way I assumed.
The first was AWS's own documentation placeholder, AKIAIOSFODNN7EXAMPLE. It didn't get flagged, which was by design. Gitleaks' default config carries a rule-level allowlist that exempts any match ending in EXAMPLE, a deliberate carve-out for AWS's own docs. This was worth knowing, but overall not alarming on its own.
The second was a key I fabricated myself, AKIAABCD1234EFGH5678. This one should have tripped the AWS-specific rule. Instead, it was only caught by the generic generic-api-key rule. This was a real signal that something was off. Digging into the why led somewhere I didn't expect: this repo's CI, using gitleaks-action@v2 with no GITLEAKS_VERSION pinned, was silently running gitleaks 8.24.3. That version predates a regex-tightening fix (merged 2025-05-27) that restricts the AWS key charset to match how AWS actually encodes tokens. Checking the Action's own source confirmed the root cause: both v2 and current v3.0.0 hardcode the same fallback version string. As a result, the Action's major version tag has nothing to do with which gitleaks binary actually runs underneath it.
The honest version of the finding isn't "we were about to leak an AWS key, and gitleaks would've missed it"; the prefix regex was actually looser, so that specific miss didn't happen. The real exposure was quieter and worse: over a year of unrelated rule improvements, new token formats for entirely different services, that this pipeline had never picked up, with nothing in place that would have surfaced it short of deliberately going and checking. I fixed it by pinning the Action to an exact tag and setting GITLEAKS_VERSION explicitly, then verified against a live CI run showing the correct version and a clean scan against the actual diff, not just "trust the new config"; I confirmed it did what it was supposed to.
The mistake that shipped anyway
The same pull request that carried this fix also carried something that should never have reached main: a deliberately vulnerable test fixture, an eval() on unsanitized input, a pinned-vulnerable dependency, a fake AWS key which was built specifically to confirm the CI jobs actually fail on bad input. That commit got merged anyway, with 2 of the 3 security jobs failing. GitHub allowed it because this repo had no required-status-check branch protection on main; nothing automated was actually gating the merge button. I only caught it by manually diffing the branch against main before treating the PR as done. Fixed with a straight revert, and all three jobs are green again in the corrected state. After the fix, I decided to leave that mistake in the write-up instead of quietly cleaning the history, because "diff the PR before you trust it" is a more convincing demonstration of the whole point than any scanner finding above it. Also, it exposed a real, still-open gap: no required checks on main at all.
When a clean SAST result isn't a gap, either
This is the same fixture, but a different tool: semgrep reported zero findings on a file containing a literal eval(userInput) call. Before assuming that was a miss, I pulled the actual community rule (eval-detected) and ran it locally against five constructed variants. The pattern that emerged: semgrep performs constant propagation before matching, so when userInput is assigned a literal string and never reassigned, eval(userInput) resolves to the equivalent of eval("literal"), which the rule's own exemption is written to treat as safe, deliberately, since a hardcoded literal isn't attacker-controlled. My test fixture's shape happened to land exactly on the one form of "bad eval" this rule is built to wave through. Not a tool gap, not a misconfiguration. It was a lesson that a hand-built adversarial test case needs the same scrutiny as a real one before its result counts as verification.
The actual takeaway
So, none of these is really about gitleaks or semgrep specifically. It's that a passing CI job is a claim, not a fact, and claims are worth checking against the thing that actually produced them: the rule source, the pinned tool version, and the diff that got merged. These all were before they get treated as evidence of anything. That standard doesn't stop at the code under test; it applies to the pipeline testing it as well. It's also the same lesson I first ran into on a much smaller scale, building this app's inventory logic in the first place. A response coming back successfully and a response coming back correct turned out to be the same distinction, just at a different layer. (That story's Part 0 of this series, if you want the origin.)
Full technical write-up with commit links: product-security-labs. Target codebase: smart-kitchen-mgmt (original team project).
Top comments (0)