If your AI coding tool changes three files and your security alerts fall from eight to five, what actually improved?
I would want to know whether the scanner changed too. Otherwise, you might congratulate a code fix for something the measurement tool did.
GitHub announced CodeQL 2.27.2 on October 9. The underlying release notes are dated October 7. I'm using that recent development from the past seven days because it offers a useful beginner lesson beyond a release headline: an analysis upgrade can change what a tool reports about the same application.
CodeQL analyzes code for patterns associated with security problems. This release improves Hapi request-input tracking in JavaScript and TypeScript. It also corrects a C# false positive involving generated Razor tag-helper attribute code. Those are two different reasons a report might change: the tool can understand a path better, or stop treating a particular pattern as a problem.
Neither outcome automatically tells you that your last app edit fixed anything.
My operating rule is simple: compare the measurement tool on unchanged code before you compare a code fix under the new tool.
If you're starting your first app, my $3 AI App Builder Starter Prompts provide a guided starting workflow. Add this comparison rule to your testing instructions so the next AI-assisted change comes with an explanation you can check.
1. Save the starting point
Before upgrading an analyzer or accepting a proposed fix, save a small evidence note. Record the exact code revision, scanner version, query suite, relevant custom queries, configuration, build mode, and whether the run completed successfully.
Also save the findings themselves. A total count is useful for orientation, but it cannot tell you whether three findings disappeared while two different ones arrived.
Think of the code revision as a labeled photograph of your repository. You need that label to compare the same application twice.
For a beginner, the important question is not whether you can memorize every tool option. It is whether you can ask your coding assistant to show what changed between the two runs without silently changing the app at the same time.
2. Hold the application still
When your environment supports it, analyze the same revision with the old and new analyzer. Keep the query suite, configuration, and build conditions comparable. Record any unavoidable differences.
That gives you a first comparison: old tool versus new tool, same code.
Then make the application fix and run the new tool again. That gives you a second comparison: old code versus changed code, same tool.
Here is a hypothetical example, not a report from one of my projects. A scanner reports an issue on revision A. After an analyzer upgrade, that issue disappears on revision A without an application edit. You can investigate the analyzer change as an explanation. You cannot honestly write that revision B fixed the issue when B did not exist yet.
Now imagine the finding remains on A under the new analyzer. You make a targeted change in B, and the same analyzer no longer reports it. That is more relevant evidence for the fix, although you still need to inspect the path and test the intended behavior.
3. Explain each changed finding
I would sort changed findings into four working explanations:
- The analyzer now recognizes a behavior it previously missed.
- The analyzer corrects an earlier false positive.
- The application change alters the reported path.
- The run or configuration changed, making the comparison incomplete.
These are investigation categories, not verdicts. Read the finding and the release notes before assigning one.
The CodeQL release's improved Hapi tracking makes a useful example. If your application uses wrappers around route registration, a newer analyzer may follow request input differently. A newly visible finding could concern code that was already there. It does not, by itself, prove that the upgrade introduced an application vulnerability.
Conversely, a removed finding deserves an explanation. A lower number can reflect better analysis rather than a safer code revision.
4. Check your custom queries separately
The release notes also describe changes to CodeQL's Go control-flow graph that can require existing custom queries to be updated.
If you maintain custom checks, ask whether those checks still compile and run as intended. A query compatibility failure belongs in the evidence note. Do not present it as a successful scan with no findings.
You probably do not need to write custom queries for your first app. You do need to recognize that the checking system has its own dependencies. An app build can succeed while a separate analysis job fails or checks something different from what you expected.
5. Give AI a comparison task before a repair task
Here is the instruction I would use:
“Compare these two security-analysis runs before editing application code. Identify the code revision, analyzer version, query configuration, and run completion state for each. List new, removed, and unchanged findings. Separate changes supported by analyzer release notes from changes supported by code differences. Mark uncertain explanations explicitly. Propose the smallest follow-up check for each uncertain finding.”
Keep secrets and private customer data out of the material you provide. A comparison note should describe the evidence needed for the task, not export your whole environment.
After the comparison, ask for one targeted fix and its verification plan. That sequence makes it harder for an assistant to produce a plausible explanation after changing several unrelated things.
The limitation: your old tool may be unavailable
Hosted tools can update automatically. GitHub says new CodeQL versions are automatically deployed to code-scanning users on github.com. You may not have a convenient old-versus-new rerun available.
In that case, preserve the previous report and compare it with a new run on the same revision where possible. Record the missing controlled comparison. Treat release notes as evidence for a hypothesis, not proof that a specific finding changed for that reason.
Do not hold important security work indefinitely for a perfect experiment. Investigate consequential findings and make needed fixes while stating what your evidence can and cannot establish.
The practical next step is small: pick one changed finding and write down the code revision, tool version, and reason you believe it changed. If you cannot name the reason, your next task is a check, not a victory announcement.
Continue from a check to a build workflow
Use the $3 AI App Builder Starter Prompts for guided first actions. For the organized path from idea through testing and publication, the $9 standalone AI App Builder From Zero e-book is the longer field manual; it excludes the separate 40-prompt workbook. The $10 Gumroad learning bundle includes both resources.
Those purchases provide respective $20, $30, and $70 buyer benefits toward Bootcamp Bundle (Review Radar) when checkout opens. One verified benefit per buyer, used once; benefits do not stack, have no cash value, and exclude refunded or disputed purchases. Previously issued benefits remain preserved.
Bootcamp Bundle (Review Radar) is coming soon, planned at $399/$999, with checkout closed. See the public preview and request waitlist updates manually by email. An email request expresses launch interest; it does not automatically register you.
You can also find me here:
Medium: https://medium.com/@marcusykim
DEV.to: https://dev.to/marcusykim
Website: https://marcusykim.com/
X: https://x.com/marcusykim
LinkedIn: https://www.linkedin.com/in/marcusykim/
Upwork: https://www.upwork.com/freelancers/marcusykim
Top comments (0)