A while back I wrote about finding security holes in my own hackathon project months after the fact. It was not a proud moment. There is a specific flavor of shame in reading code you wrote, out loud, going "who hurt this person," and slowly realizing it was you.
So this time I flipped it around. Instead of finding bugs by accident later, I would put them in on purpose now, and build a system whose entire job is to catch me doing it. Meet Break&Detect: a small Flask REST API that I made bad on purpose, wrapped in a CI pipeline that yells at me every time I try to ship something stupid.
Here is what I learned, which mostly comes down to one thing. The robots are good. They are not that good.
Part one: building the trap
The idea was a security gate in GitHub Actions. Every push runs a lineup of scanners, and if any of them find something HIGH or CRITICAL, the build goes red and the merge is blocked. No "we will fix it later." Later is where security bugs go to live forever.
The lineup, eight stages of it:
Gitleaks, for secrets I definitely did not mean to commit (I did)
Bandit and Semgrep, for the classic "you built a SQL query with an f-string" energy
Trivy, twice, once on the dependencies and once on the container image
Checkov, for the infrastructure config
OWASP ZAP, actually poking the running app like a bored attacker would
Every finding gets pushed up to the GitHub Security tab as SARIF, so instead of squinting at raw logs you get a tidy list of your sins in the UI.
I will be honest about the part nobody puts in the README. Wiring up GitHub Actions is about 20% writing YAML and 80% pushing a commit called "fix ci" and watching it fail for a reason that makes no sense. My commit history for that week reads like a man slowly losing an argument with a computer. But once it worked, it worked, and watching the build go red the first time it caught a real bug felt genuinely great.
Part two: why I did not stop there
Here is the thing about scanners. They are pattern matchers. They are excellent at "this looks like a hardcoded key," or "this function call is a known injection sink," or "this dependency has a CVE." That is real value and I am not knocking it.
But a scanner does not understand what your app is for. It does not know that note number 5 belongs to a different user than the one asking for it. It cannot tell that your login endpoint will happily accept ten thousand password guesses without blinking. Those are logic bugs, and logic lives in the gap between what your code says and what your app actually means.
So once the pipeline went green on the easy stuff, I put on the other hat and pentested the thing by hand. Burp Suite, sqlmap, Hydra, and a truly unreasonable amount of curl, with the OWASP testing guide as the map. The goal was simple. Find what the robots could not.
I found eight things. A couple were exactly what you would expect. A couple were more fun.
The one that actually scared me
The best bug was not one bug. It was two boring ones holding hands.
Bug one: the JWT signing key was hardcoded, sitting right there in the source as a plain string. Bug two: tokens were set to never expire, and the code that checks expiration had it explicitly switched off.
On their own, each is a "yeah yeah, fix it eventually" finding. Together they are a skeleton key. Anyone who can read the repo grabs the signing key, forges a token for literally any account including admin, and that token works forever. No password, no login, no expiry to wait out. Just permanent access to everyone, minted by hand.
And because self-registration was wide open, there was a second, totally separate way in. Register a throwaway account, then use a UNION-based SQL injection in the search endpoint to pour the entire users table, password hashes and all, back through what was supposed to be a search of your own notes. Two doors, same room, and the room has everybody's data in it.
This is the part scanners get half right. Gitleaks did flag the hardcoded key. But "here is a hardcoded key" and "here is how this key becomes full account takeover for every user in two steps" are different sentences, and only one of them makes a person actually go fix it before lunch.
The bugs the robots could not see
Two findings were pure logic, and no scanner was ever going to catch them.
The first was textbook BOLA, which is a fancy way of saying the notes endpoint looked up your note by its ID number and never once checked whether that note was actually yours. Want to read someone else's private note? Add one to the number in the URL. Want to delete it? Same move, different verb. The code worked perfectly. It just trusted everyone completely, which is adorable in a person and a disaster in an API.
The second was my favorite, because it is so human. The app had a "fetch this URL" feature, which is SSRF bait and I knew it. So past me had written a config file with a responsible little security section: an allow-list of schemes, a flag to block private networks, the whole adult setup. Great. Except the actual code never read that config. The protection existed purely as a vibe. I had written the security policy and then forgotten to plug it in, so the endpoint would cheerfully go fetch the cloud metadata address at 169.254.169.254 and hand back whatever it found.
No scanner catches that, because on paper everything looks fine. You have to read what the code does versus what it pretends to do. That gap is where the interesting stuff always hides.
The one I got wrong, and kept anyway
One of my eight findings turned out to be a false alarm. Trivy flagged a known CVE in a pinned dependency, I sat down to write it up, then checked the actual version and realized it had already been patched in an earlier commit. Not exploitable. No impact.
I kept it in the report anyway, marked "not reproducible," with the verification steps written out.
Why keep a non-bug in a security report? Because a report that only contains wins is not a report, it is a highlight reel. Showing the thing you checked and honestly ruled out is kind of the whole point. It tells whoever reads it that the other seven findings were verified the same careful way, not just copied out of a scanner and dressed up in a suit. The false alarms are how you earn trust in the real ones.
The satisfying part
Then I fixed them. Parameterized the query, moved the key into an environment variable and rotated it, made tokens expire and added rate limiting, put an ownership check on the notes, actually wired up the SSRF config I had been ignoring, escaped the XSS, killed debug mode, added the security headers.
Then I re-ran every single exploit against the fixed branch, one at a time, to confirm each one was properly dead and not just hidden. The pipeline went green. The forged-token trick started returning a polite 401. The BOLA attempt returned a 404 that no longer even admits the note exists.
Watching your own attacks stop working is a weirdly lovely feeling. Ten out of ten. Would break my own app again.
The actual takeaway
If you want it in one sentence: automated scanners are smoke detectors, and manual testing is the person who walks over and checks whether the stove is actually on. You want both. The pipeline caught the bugs that look like bugs. I had to catch the ones that only look wrong once you understand what the app is trying to do.
The whole thing is on my GitHub, pipeline and full report included. My one piece of advice, hard earned, is to write the config that protects you and then also remember to read it. Ask me how I know.
Top comments (4)
Great experiment. Intentionally planting vulnerabilities is a practical way to evaluate what AI security tools can—and can't—detect. Results like these are a good reminder that benchmark performance doesn't always translate to comprehensive coverage in real applications.
The takeaway isn't that AI is ineffective, but that it works best as part of a layered security process alongside code reviews, static analysis, testing, and human expertise. Understanding the gaps is just as valuable as measuring the successes. Great write-up!
Yes truly this is the takeaway, AI and all the other tools are absolutely amazing for many above the top errors and vulnerability finding but to dig deep humans have to interfere and use their own hands!!
Thank you so much for the kind words, glad you liked it!!
Some comments may only be visible to logged-in visitors. Sign in to view all comments.