Last month I merged a pull request where roughly 80 percent of the diff came out of an AI agent. The tests passed. The build was green. I approved it in about four minutes. That night I could not stop thinking about one line, a file path built by string concatenation from user input, and whether I had actually verified it was safe or just believed it was.
I am a Senior Software Engineer II at BS23 in Dhaka, and I have written Spring Boot for six years. Over the last year I have also been running my own AI agent infrastructure, the same stack that publishes my articles. The uncomfortable truth of 2026 is that these two worlds have merged on my machine. The agent writes code. I review it. And the entire safety of that arrangement rests on my review catching things I did not write and did not fully trace.
So last week I stopped trusting my eyeballs and ran an experiment on my own repositories: GitHub's CodeQL and Semgrep, side by side, watching every commit. Full disclosure before anything else: I am a GitHub Advanced Security customer, and this article is not sponsored by anyone. Here is what a week of listening to two scanners argue taught me about shipping AI code without fear.
The numbers that forced this experiment
AI code is now a large share of everything, and it is not secure by default. The measurements I kept coming back to while setting this up:
- Over 500 million AI-generated commits have landed on GitHub, from 1.2 million contributor and agent accounts, per Sourcegraph's deep research report. That is not a future trend. That is the current state of the world's code.
- In real production repositories, 25 to 46 percent of committed code is AI-generated, according to a July 2026 Forge analysis of 486 of them. It found engineers who use agents each contribute more than twice the volume of AI code, about 2,100 lines per month, and that startup repos run near 60 percent.
- 45 percent of AI-generated code samples fail security tests, introducing OWASP Top 10 vulnerabilities, in Veracode's testing of over 100 large language models. Their March 2026 update found the pass rate flat at roughly 55 percent across the whole testing period, even as the models got demonstrably better at writing code.
- The CVE counter is spinning. The Cloud Security Alliance's tracking recorded 6 CVEs attributable to AI-generated code in January 2026, 15 in February, and 35 in March. A near sixfold jump in two months.
One number from that CSA note made me genuinely uncomfortable: Claude Code accounts for 27 of the 74 confirmed AI-attributed CVEs, partly because it leaves identifying signatures in its commit messages. The uncomfortable takeaway is not about one tool. It is that the attribution we can see is probably a fraction of the total, and nobody is marking the rest.
The Stanford study is the one I think about most, though. Researchers found developers who used AI assistants wrote less secure code in 4 out of 5 task types, and, here is the part that should worry every reviewer, those developers reported feeling more confident about the security of their code. I recognized myself in that. Confident and wrong is the exact failure mode of a fast approve button.
The setup, in plain terms
Both tools are free to start, which is the only reason I could run them side by side without asking anyone for budget.
CodeQL is GitHub's semantic analysis engine. It builds a relational database out of your entire codebase and then runs queries against data flow: where does this input originate, and can it reach a dangerous sink. Its default setup is free on public repositories and included in GitHub Advanced Security for private ones, which is how I run it.
Semgrep is the open-source scanner that reads code like a very smart grep, matching patterns against the syntax tree with rules written in YAML. The Community Edition is free, has thousands of community rules, and supports over 30 languages.
I enabled CodeQL's default setup on my repos, added the Semgrep GitHub Action with its recommended ruleset for Java and TypeScript, and then did what I was going to do anyway: let my agents write code. For one week, every AI-authored diff got scanned by both before I merged anything. The question was simple. When my review misses something, which of these two actually catches it?
The finding that changed how I see the two tools
Semgrep reads code. CodeQL reads programs. That sentence sounds like marketing until you watch them handle the same real diff.
Mid-week, my agent produced a change in a Node.js service that took a query parameter, passed it through two helper functions, and eventually interpolated it into a shell command. Classic command injection, and exactly the kind of thing a fast human review skims past because each function looks harmless in isolation.
Semgrep flagged a weak-pattern match nearby but stayed silent on the actual injection, because the tainted value crossed a function boundary, and Community Edition taint analysis is intraprocedural. It tracks data within a single function. Across functions, its visibility ends.
CodeQL caught it, flagged the full path from HTTP input to the shell sink, and suggested the fix. That is the structural difference. Independent testing has quantified it too: one 2026 benchmark fed 164 known CVEs to both tools under default settings and CodeQL recalled 33.6 percent versus Semgrep OSS at 23.2 percent, with both numbers embarrassingly low but CodeQL consistently ahead on deep, multi-step flows like SSRF where Semgrep's simple pattern matching scored zero.
So CodeQL wins, right? Not so fast, and this is the part nobody tells you.
Semgrep is faster by an order of magnitude, and speed changes behavior. CodeQL has to build a database of your codebase before it can query anything, which takes minutes. Semgrep returns results in seconds. In practice this meant Semgrep ran on every commit, and CodeQL results arrived on the pull request. Feedback that arrives in seconds gets read; feedback that arrives after a coffee break gets batched and skimmed. That is not a flaw in CodeQL, it is just what its depth costs, and it shaped how I actually used each one.
The other gap runs the opposite direction. CodeQL's language coverage is narrower and historically weakest at the edges: C++, Swift, Rust. Semgrep's flat YAML rule model supports over 30 languages, including ones CodeQL does not cover, and writing a custom rule for a project-specific dangerous pattern took me minutes. For anything exotic or internal, Semgrep is the only one of the two that can be taught quickly.
What a week of dual scanning actually caught
The volume was lower than the headlines promised, and that surprised me most. My agents produce maybe 40 pull requests in a typical week across my repos. Over the experiment week, the two scanners together surfaced findings I had not caught in 3 of them. That is 7.5 percent, not 45 percent. The difference, I think, is that the scary statistics come from asking models to write vulnerable patterns on demand, while agentic workflows with established repos, lint configs, and existing patterns anchor the model to safer code.
But 3 pull requests out of 40 is not zero, and here is what was in them:
- One command injection path in Node.js, the one described above. CodeQL's catch. Exploitable in principle, missed by me, missed by Semgrep CE.
- One hardcoded credential, a token the agent pasted into a config default to make a test pass. Semgrep's secrets rules caught it in seconds. CodeQL's default setup does not surface that class as prominently, since secret detection is a separate GitHub feature.
- One dependency-level problem, a package the agent upgraded to a version with a known advisory. Honestly, neither SAST tool owns this. Dependabot caught it. Lesson noted: SAST does not replace software composition analysis, and AI agents touch dependencies constantly.
The false positive rate was where my week got interesting. CodeQL's findings were almost all real, but a few were the kind of theoretical flow that takes twenty minutes to verify manually. Semgrep's default rules produced more noise overall, but the noise was cheap to triage because each finding came with a readable rule and a one-line explanation. I disabled two noisy rules and the signal got good fast. With CodeQL you tune less but investigate longer.
The verdict: it is not X or Y, it is X then Y
If you can run only one and your code is on GitHub, run CodeQL. Depth on data-flow vulnerabilities is the thing that matters most for AI code, because the bugs agents introduce are rarely simple patterns. They are plausible-looking values traveling through innocent-looking helpers. That is CodeQL's home turf, and its free tier on public repos makes it the default for anything open source.
Add Semgrep when speed, language breadth, or custom rules matter. For me that meant keeping it as the fast per-commit gate. If your stack includes languages CodeQL handles weakly, or your team writes its own rules, Semgrep earns its place immediately.
Here is what I would tell anyone in my position, in the order I would do it:
- Turn on CodeQL default setup today. It is a checkbox in repo settings for public repos, and the results from week one will recalibrate how much you trust your review.
- Add Semgrep with its recommended ruleset as a required check. Expect noise for the first two days, then tune.
- Keep Dependabot on. Agents change dependencies, and no SAST tool sees a bad upgrade coming.
- Never let an agent-generated diff skip human review, but do not let review carry the security load alone. The Stanford confidence finding applies to all of us, and I proved it on myself with a four-minute approve.
- Treat secrets scanning as mandatory. The single worst thing my agent did all week was paste a real-looking token into a config file, and it did it casually, to make a test pass.
The fear I started the week with was the wrong emotion. The right one is something closer to professional humility: the machines writing my code are neither safe nor malicious, they are just indifferent, and indifference scales. Two free scanners arguing over every diff is a small price for the version of my job where I still merge agent code and still sleep.
I write about AI infrastructure, developer tools, and backend engineering every week. Subscribe, it is free, and next week I am doing the same experiment on the AI agents' infrastructure code, which is where I expect the results to get worse.
Do you scan AI-generated code differently from human code, or has your team not drawn that line yet? Tell me what your setup looks like, I am genuinely collecting approaches for a follow-up.
Sources: Sourcegraph AI-generated code research, Forge analysis of 486 production repositories (July 2026), Veracode "AI Models Are Still Failing Security" (March 2026), Cloud Security Alliance research note on the AI-generated CVE surge (2026), and the Ansede Static SAST benchmark of 164 CVEs (July 2026).
Top comments (1)
Running two scanners against the same diff and keeping them in disagreement is a much better use of them than either alone. In our stack the pattern was the same: the tools rarely fight about the obvious findings, they fight about path handling and injection surface, which is exactly where AI-generated code is weakest - string concatenation into file paths was literally our first near-miss too.
How did you close the loop on false positives? We ended up triaging by whether a finding had a reachable source, not by severity label, or the noise buries the two findings that matter.