Checking the code you wrote: static analysis

This is part 6 of the series "Protect Your Own Service Yourself — Self-Defense Security in the Age of AI." Do the checks you can yourself; when it's hard, EarthLink Network can run the assessment for you, and fix the gaps you can't fix on your own.

Series contents

1. Anyone can attack now, with AI — check your own site first
2. What attackers now hold: when AI runs the attack itself
3. Before you start: the whole picture of checking, and how to test without breaking production
4. Testing from the outside (1): setup and the first run
5. Testing from the outside (2): attack tools and legitimate scanners find different things
6. Checking the code you wrote: static analysis (this article)
7. Checking the parts you depend on: libraries and packages (coming soon)
8. Checking your infrastructure settings for gaps (coming soon)
9. Collect every result in one place and decide what to fix first (coming soon)
10. How to fix what you found: three ways (coming soon)
11. When you cannot fix it yourself: a maintenance contract (coming soon)
12. Keep checking automatically every day — from building to protecting (coming soon)

The short version

We scanned our own service's code with a free static analyzer and its security rule set. Zero findings.

Then we ran the same code through a different rule set, the JavaScript one. One finding: the line that decides where to send a user after login.

And the flaw behind that line, a check that let /\ slip through, was not caught by any rule in the Semgrep rule sets we ran. Our AI code reviewer, CodeRabbit, caught it when we ran it on the fix. (It has been fixed.)

That is the whole lesson of this part. Zero findings does not mean safe. Different ways of checking find different weaknesses. Layer them, check every hit for false positives, and run the checks again on a schedule.

Static analysis has a second job, too: finding secrets, like API keys and passwords, written into the code. When we searched our git history with Gitleaks, an AWS access key committed earlier was still there, even though it had been deleted from the file. It turned out to be deactivated already. Finding it is covered here. Revoking it and scrubbing the history properly is Part 10.

The rest of this part has three steps:

1. Scan. Three ways to check code, what happened on our own service, and how to search for leftover secrets.
2. Sort out false positives. Where to look, and what to look at, to tell a weakness from noise.
3. Run it again on a schedule. So one zero is never the end of it.

1. Scan

What static analysis is

Static analysis reads your source files one by one and flags code written in a risky way. For each hit you get the file, the line, and an explanation.

Parts 4 and 5 compared testing a running service from the outside with an AI tool that reads the code (Strix). This part is about code-reading checks themselves. Compared with outside testing:

• It checks every file by the same standard. Outside testing only reaches the entry points the world can see. Reading the code also covers features not yet on screen and admin-only features.
• It is cheap and fast. Many tools are free, and thousands of files take minutes.
• It cannot see what only shows up at runtime. Whether the server sends the right security headers, for example, is a job for outside testing.

You need both. This part covers the code side.

Three ways to check

Static analysis tools fall into three groups, by how they find weaknesses. This matters more than the brand names, because a different method finds different things.

• Match against rules. The tool carries hundreds of rules ("this pattern is risky") and compares every file against them. A bundle of rules is called a rule set. Great at sweeping for known bad patterns; blind to anything no rule describes. Free options: Semgrep Community Edition and SonarQube Community Build.
• Follow the data. The tool tracks a value from where it enters, across files, to where it is used. That catches things like "a user-controlled value reaches a database query unchecked," with fewer false alarms than rules alone. Free options: Semgrep's free plan (with an account) and GitHub CodeQL for public repositories. SonarQube has this too, but only on paid plans (Team or higher on SonarQube Cloud).
• Let an AI read it. An AI reviews the code the way a colleague would, and can notice reasoning errors no rule captures, such as "this check isn't enough." Strix, from Parts 4 and 5, is this kind of tool. For everyday reviews we use CodeRabbit. It reviews every pull request automatically and comments on the lines it is worried about. You can also run it from the command line on work in progress.

All of them have a free starting point. The table near the end has the details.

Same code, two rule sets

Here is what happened on promptflow, our own service. We scanned a copy of the code from before this weakness was fixed. No production system was touched.

Semgrep installs with one line, python3 -m pip install semgrep. We pointed it at the app's source folder with the security rule set, p/security-audit.

2,016 files in about two minutes. Zero findings.

Then the same code with the JavaScript rule set, p/javascript.

One finding. The flagged line decides where to send the user after login. In plain words, Semgrep is saying: "The destination comes from a value a user might control. If an attacker can set it, they can bounce your users to a fake site. Check it before you use it." This is called an open redirect, and it is a classic way to lure people onto a fake login page.

That rule simply is not in the security rule set. Same method (rule matching), same code; switching rule sets changed what was found.

A different method finds what no rule covers

Is the line dangerous? Following the code, the destination is checked first, in a function in another file. The check keeps the destination on the same site (paths like /settings) and replaces anything starting with https:// or // with the home page.

But the check had a flaw. It blocked //, yet let /\ (a slash followed by a backslash) through. Browsers sometimes treat a backslash in a URL like a forward slash, so /\ can be read the same way as //: go to another site. Get someone to click a crafted link and log in, and the code allowed sending them elsewhere afterwards.

Semgrep did not find that flaw. While fixing the line, we ran CodeRabbit's command-line review on the change, and it pointed out /\. None of the Semgrep rule sets we ran describes that pattern. Rule matching and AI review find different things. After that review, we changed the check so values starting with /\ also go to the home page.

The step-by-step trace through the code is in the appendix.

Why zero is not safe

The first scan's zero only meant "no code matched these rules." Three reasons it is not a clean bill of health:

• The rules are limited. Only 23 rules ran. That is not every risky pattern in the world.
• The free engine stays inside one function. Per Semgrep's docs, the account-free Community Edition does not follow a value from one file to another.
• Some files were cut short. The JSON results recorded timeout warnings on 22 files. The on-screen summary still said "Scan completed successfully," so you only notice if you open the result file.

So you layer methods: add rule sets, add a tool that follows the data, add AI review. One method's zero can be another method's finding.

Secrets left in code, including git history

The second job of static analysis is finding secrets, like API keys and passwords, written directly into code.

This is a different kind of risk. Everything above was about whether the running site can be broken into from the outside. A leftover secret never appears in any response. You could send a million requests and never see it, and on its own it does not let anyone in from the outside.

The danger starts when the repository itself ends up in someone else's hands. Say a GitHub account or a developer's laptop gets compromised. Any key in the code can then be used to run up your cloud bill or pull your data.

And here is the part people miss: git keeps every past change. If you commit a key and delete it in the next commit, the old commit still has it. Anyone with the repository can pull it back out. That is why you search the history, not just the current files.

Gitleaks is a single binary from its GitHub releases page. We scanned the full history of the promptflow repository.

7,633 commits in 1 minute 26 seconds. Two hits, on two adjacent lines of the same file: one shaped like an AWS access key ID, the other like its matching secret.

Always pass --redact. Without it, the secret is printed to your terminal and saved in the report file, and the moment you share that file you have leaked the key a second time.

2. Sort out false positives

Rule-based tools flag things "just in case." Not every hit is a weakness, so check each one.

Code findings

Start at the flagged line and trace where the value comes from, and whether it is checked before use.

What tracing shows	Verdict	Next step
The value never comes from outside	False positive	Note why, mark it as a false positive
It comes from outside but is checked before use	False positive (the tool can't see the check)	Note where the check is, mark it
It is checked, but the check falls short	Weakness (minor)	Fix the check
It comes from outside and is never checked	Weakness	Fix first

Our Semgrep finding looked like row 2 at first and turned out to be row 3. The tool's output alone cannot tell you which row you are in. Only a person or an AI tracing the code can.

Leftover secrets

For a leaked secret, check three things: is it a documentation sample, is it a made-up test value, and does it still work. The report hides the value, so we opened the flagged commit locally. (The value appears nowhere in this article or in our notes.)

• Neither a sample nor a test value. AWS access key IDs are 20 characters starting with a fixed four-letter prefix. The line matched, and it was not the documentation sample that ends in EXAMPLE. A key had been pasted straight into a small local test script.
• Deleted from the file, still in git history. The current files no longer contain the key. The old commit still had it.
• Already dead. A read-only AWS API call showed the key as Inactive, with a newer key issued for the same user. It had been replaced and switched off.

When you find a leaked key, the first question is always "does it still work?" How to revoke it and how to scrub git history properly are Part 10.

Mark false positives so they stop coming back

Once you know a hit is a false positive, mark it, or the same noise comes back every run and buries the new findings.

• Semgrep: add // nosemgrep: <rule id> on the flagged line, with your reason next to it.
• Gitleaks: add gitleaks:allow on the line for a test value, or put the finding's fingerprint (an ID string Gitleaks assigns to each finding) in a .gitleaksignore file.

Always write the reason. A mark without a reason can just as easily hide a weakness. And keep each exclusion to a single line: excluding whole test folders means a key pasted there will be missed too.

3. Run it again on a schedule

A zero today is a zero for today's code with today's method.

• The code changes every day. New code can bring new weaknesses.
• The rules grow. When a new attack technique becomes known, rules to catch it get added. Yesterday's clean code can fail today's rules.
• Secrets slip in at any time. A key pasted in "just to test" and committed by mistake happens to everyone.

So turn it into a routine. We run the checks at three points:

• On every pull request. Semgrep with the JavaScript, Python, and security rule sets, plus Gitleaks and Trivy, which checks dependencies (Part 7). CodeRabbit reviews the code too. New high-severity findings block the merge.
• Right after merging. Semgrep, Gitleaks, and Trivy run again.
• Every day. GitHub Actions runs Semgrep, Gitleaks, and Trivy.

Nobody does anything by hand each morning. Set it up once and the machine checks every day. Results go to the collection point in Part 9, and Part 12 shows the daily runs.

What is free

Everything here has a free starting point. Prices are from each vendor's official pricing page.

Tool	Method	Free tier
Semgrep Community Edition	Rules	Unlimited (open source)
Semgrep free plan	Rules + data flow	Up to 10 repositories and 10 contributors
SonarQube Community Build	Rules	Unlimited if self-hosted; data-flow detection needs a paid edition
SonarQube Cloud Free	Rules (data flow needs Team)	Private code up to 50,000 lines
GitHub CodeQL	Data flow	Public repositories only; private needs a GitHub Team plan or higher plus $30 per active committer per month
CodeRabbit	AI review	Public repositories get full pull request reviews free; private repositories get pull request summaries plus IDE/CLI reviews (3 per hour per user)
Gitleaks	Secrets	Unlimited (open source)

Our repositories are private, so we pay for CodeRabbit's pull request reviews, and we use neither CodeQL nor GitHub's secret scanning ($19 per committer per month on private repositories). We run free Semgrep and Gitleaks every day instead. Part 3 compares pricing in more detail.

Next

Part 7 checks the parts you depend on. The code you write is a small share of what you ship; most of it is third-party packages. We will check those packages for known vulnerabilities, with commands and screens again. This is the check that usually turns up the most findings.

Appendix

Tracing the Semgrep finding through the code

The flagged line 53, next to where its value comes from:

1. Line 53 navigates to result.returnTo, the post-login destination. This is what Semgrep flagged.
2. That value is built by a function in another file, resolveSilentSsoCallbackError.
3. At line 557, that function compares the state value in the URL with the one this browser saved when it started the login. No match, no action. An attacker cannot inject a value this way.
4. At line 565, the destination goes through sanitizeReturnTo, which keeps it to a path on the same site and replaces anything starting with https:// or // with /.

Because the check lives in another file, the free Semgrep engine, which does not follow values across files, cannot see it.

But the check in step 3 only stops someone swapping in their own state. The destination can be set earlier, by another route:

1. The login entry page, /sso, accepts a returnTo value in the URL as the post-login destination.
2. That value goes through sanitizeReturnTo. Anything starting with /\ slips through.
3. The user's own browser saves it into state.
4. After login, state matches, because the browser saved it itself, and the page navigates to the /\ destination.

That final navigation happens at line 33 of the same file, not the flagged line 53. Semgrep flagged only one of the two places that can send the user to another site.

What the commands mean

• semgrep scan --config <rule set>: which rule set to use. p/security-audit and p/javascript are fetched automatically from Semgrep's public registry.
• --metrics=off: do not send usage statistics to Semgrep.
• --json --output <file>: save machine-readable results. Things the screen does not show, like timeout warnings, are kept here.
• gitleaks git .: scan the full history of the current repository. Use gitleaks dir to scan only the current files.
• --redact: replace found values with REDACTED, on screen and in the report.

---

In this series, in an age when attack tools are free and one click away for anyone, I show step by step how to check your own service yourself and close the gaps before someone else finds them. The target is always our own service; the series does not cover attacking anyone else's site.
If running the checks yourself is hard, EarthLink Network can do the assessment for you. Found a gap you can't fix? We'll handle the fix. Feel free to reach out.
If this was useful, please like and follow.

Our products are listed at https://www.eln.ne.jp/products

---

EarthLink Network builds its own products to run the whole company on AI — each one born from a need along the way. See the full index of what we build:

→ The 18 products EarthLink Network builds in-house

For more about the company and each product, visit www.eln.ne.jp.