Before you read: we are not security experts. We are a small team learning security tooling by testing it directly and writing down what happened. This is a field note, not a best-practices guide. If something here is wrong, corrections in the comments are welcome.
TL;DR
We put 10 common vulnerabilities into a React and TypeScript app on purpose, then ran the free, zero-config Semgrep (--config=auto) against them. It reported 3 of the 10. The gap is not a defect in Semgrep. It reflects what free static analysis is built to find and what it is not. The details are below.
[IMAGE PLACEHOLDER, COVER] A cover image goes here. The scorecard graphic or a screenshot of the GitHub Actions run both work.
Why we ran this
TL;DR: We wanted a measured answer to one question. On its own, how much does a free SAST scan actually find?
"Add SAST to your CI" is common advice. The part that rarely gets measured is what you get from it when you pay nothing and write no custom rules. Instead of guessing, we set up a small test:
- A Security Operations Dashboard built with React 18, TypeScript, and Vite.
- A folder,
src/security-test-cases/, holding 10 deliberately vulnerable files, one per weakness. - A GitHub Actions workflow that runs
semgrep scan --config=autoon every push.
Then we read the CI logs and recorded the results.
How Semgrep works
TL;DR: Semgrep parses your code into a tree (an AST) and matches rule patterns against that tree. It does not run your program. If no rule describes a given pattern, that pattern is not reported.
The same idea as a text diagram:
Source code -> Parser -> AST -> Match rules -> Findings
(.ts/.tsx) (a tree of (~1074 (file + line
your code) patterns) + rule id)
A few things follow from this design:
- It is fast and cheap to run. There is no compile or execution step, so a full repo scans in seconds, which suits CI.
- It handles bad code that is present. A call like
crypto.createHash('md5')has a stable shape that a rule can match. - It struggles with correct code that is absent. The missing presence of an authorization check is hard to express as a pattern.
The 10 cases we planted
TL;DR: One file per weakness, each mapped to its CWE identifier. All are isolated and never imported by real application code.
| # | Vulnerability | CWE | Risk in one line |
|---|---|---|---|
| 1 | Hardcoded credential | CWE-798 | Secrets in git history are hard to rotate and are exposed to anyone with repo access |
| 2 | IDOR | CWE-639 | Trusting a user-supplied ID lets one user read another user's data |
| 3 | Sensitive info leak (logs or localStorage) | CWE-532/312 | Tokens in logs or localStorage are easy to read |
| 4 | Broken crypto (MD5) | CWE-327 | MD5 is unsafe for password or integrity use |
| 5 | XSS via dangerouslySetInnerHTML
|
CWE-79 | Rendering raw user HTML lets a script run in another user's session |
| 6 | SQL or NoSQL injection | CWE-89 | String-concatenated queries let an attacker rewrite the query |
| 7 | Insecure deserialization or unsafe eval
|
CWE-502 | Evaluating untrusted input allows code execution |
| 8 | CSRF (no token) | CWE-352 | Cookie-only state changes can be triggered by another site |
| 9 | Open redirect | CWE-601 | Redirecting to an unchecked URL enables convincing phishing |
| 10 | Improper error handling | CWE-209 | Raw stack traces reveal internal paths and logic |
The setup did not work on the first try
TL;DR: The pipeline broke several times before it produced results. Each failure had a specific cause. The sequence is below.
Attempt 1 Failed. We used a flag that does not exist, `--soft-fail`.
CI stopped with "unknown option". Lesson: check the CLI reference, not memory.
Attempt 2 Failed. The SARIF upload hit "Resource not accessible by integration".
Cause: missing `permissions: security-events: write`. Tokens are read-only by default.
Attempt 3 Partial. On a private repo, GitHub code scanning (Advanced Security) was unavailable.
We made the upload non-blocking. The Security tab is not free on private repos.
Attempt 4 Not useful. `--sarif --output=file` suppressed the readable report.
The log showed only a count, no paths. We split it into two steps.
Attempt 5 Worked. Findings appeared in the log with path, line, and rule id.
Most of our time went into configuration rather than security: flags, permissions, and output formats. That is worth planning for.
Results
TL;DR: The first round, with generic mock code, reported 1 of 10. After we rewrote the fixtures to resemble real framework code, it reported 3 of 10. The remaining 7 stayed unreported across six different rule packs.
| # | Vulnerability | Round 1 (generic) | Round 2 (realistic) |
|---|---|---|---|
| 4 | Broken crypto (MD5) | ✅ | ✅ |
| 7 | Unsafe eval()
|
❌ | ✅ |
| 9 | Open redirect | ❌ | ✅ |
| 1 | Hardcoded secret | ❌ | ❌ |
| 2 | IDOR | ❌ | ❌ |
| 3 | Sensitive info leak | ❌ | ❌ |
| 5 | XSS | ❌ | ❌ |
| 6 | SQL or NoSQL injection | ❌ | ❌ |
| 8 | CSRF | ❌ | ❌ |
| 10 | Stack-trace leak | ❌ | ❌ |
The unplanned finding
TL;DR: The same scan reported 3 real issues we did not plant, which is useful evidence that the tool works on real code.
- A mutable GitHub Actions tag (
@v4rather than a pinned SHA), which is a supply-chain risk. - An incomplete-sanitization issue in our actual
metricsApi.ts. - A ReDoS risk from a non-literal RegExp in our actual
emlParser.ts.
The 7 misses are not a sign of a broken scanner. On production code, it reported genuine problems.
Why 7 were not reported
TL;DR: Three structural reasons, none of them a bug. Pattern matching has limits.
It matches shapes, not runtime behavior
A static matcher cannot follow a value from an HTTP request through several functions to a dangerous call unless a rule already describes that flow. Deeper cross-file taint tracking is mostly a paid feature.
Missing-check bugs are hard for generic rules
IDOR and CSRF are not cases of bad code being present. They are cases of expected code being absent, such as a missing ownership check or a missing CSRF token. A generic community rule has no way to know what your application's auth is supposed to look like.
Secrets detection is a separate tool
Finding hardcoded keys relies on entropy analysis and provider-specific patterns, closer to TruffleHog or GitGuardian than to AST matching. Our AWS and Azure style placeholders were not reported even by the dedicated secrets pack, because that capability is a different product.
What static SAST handles well, and what it does not
| Where static SAST does well | Where it does not |
|---|---|
Known dangerous functions such as eval and md5
|
Business-logic flaws such as IDOR and broken access control |
| Risky configuration such as mutable CI tags | Cases where a required check is simply missing |
| Syntactic anti-patterns | Data flow across multiple files, without paid taint tracking |
| Fast, cheap, consistent at scale | Secrets, which need a dedicated scanner |
Is it still worth running? Yes, as a first layer
TL;DR: A clean Semgrep run means no known pattern matched. It does not mean there are no vulnerabilities. Use it as the fast first filter, not the only control.
What we would suggest for a small team:
- Keep Semgrep OSS in CI. It is free and reported 3 real issues for us, so there is no reason to remove it.
- Do not read "0 findings" as "secure". It means no generic pattern matched.
- Write a small number of custom rules against your own auth helpers, for example flagging any
/api/handler that does not callrequireOwnership(). A handful of targeted rules will find more of your IDOR and CSRF cases than any generic pack. - Add a dedicated secrets scanner. Do not rely on
--config=autofor keys. - Keep code review in the loop. A reviewer asking whether an endpoint checks ownership catches what a free static tool will not.
What we took away from it
TL;DR: The value of a tool comes from knowing where it stops.
- A green check is a starting point, not a guarantee.
- Much of the work is configuration, not analysis. Plan for that.
- Free SAST, custom rules, a secrets scanner, and code review work together. No single tool covers all of it.
- Measuring your own coverage, such as this 3 of 10, is more useful than a vendor's claim.
If you are early in learning security, a small test like this is worth building. Testing the tool directly taught us more in a day than reading about it would have.
We are still learning, so corrections and additions in the comments are welcome.
Top comments (5)
"The value of a tool comes from knowing where it stops" — that sentence is worth more than most security tools' documentation.
But the part I'm actually taking home is your method, not your findings. Planting known-bad seeds and measuring detection is testing the tester, and it's embarrassingly rare. I build internal tools for a hospital (physical therapist by trade, not a security person either), and I've assembled my own audit layer — a static scanner over our codebase plus an AI reviewer that reads the code for auth and injection issues. I have never once seeded it with known vulnerabilities to learn its actual catch rate. I've been trusting a smoke detector I never held a match under. That changes this month, with roughly your list — it's a good spread of pattern-shaped vs. logic-shaped bugs.
Your caught/missed split also matches my experience from the opposite direction: the three Semgrep caught are exactly the kind an AI code reviewer doesn't need help with, while several it missed (IDOR especially) are where an AI reading the code with context earns its keep — IDOR isn't a pattern, it's an absence, and absences don't grep. Static filter first, context reader second, human on top. Neither layer replaces the others; each one stops somewhere different.
Also respect for reporting the accidental finds in your real code. The honest number — 3 of 10 — will help more teams than a vendor benchmark ever did.
Really appreciate this. Your smoke detector analogy perfectly captures why we ran the test.
I’d be genuinely interested to see how your static scanner and AI reviewer perform once you seed the known vulnerabilities, especially on issues like IDOR. Thanks for sharing this.
Your comment was the nudge — I ran it that same week. Seeded 10 into a throwaway PHP project; first pass caught 7 of 10. And the 3 it missed were exactly your split: hardcoded secrets, reflected XSS, and open redirect — all the "meaning" bugs, none of the "pattern" bugs. Regex catches the shape; it's blind to intent. I added rules for the three, re-ran, got 10/10.
But here's the part that made me laugh, because it happened to you too: the moment I added the hardcoded-secret rule, it fired on real code — three live API keys I'd have sworn weren't there, in programs I'd already "reviewed." Your accidental finds weren't luck. Seeding the detector is quietly also an audit of the thing you're detecting on.
On IDOR specifically — you picked the exact one I couldn't seed, and I think that's the lesson hiding in your post. IDOR isn't a pattern, it's an absence: a missing ownership check between "this ID" and "this user." There's nothing for a regex to match, because the bug is the thing that isn't there. That's precisely where a static scanner stops and a context-reading AI reviewer has to start — it has to notice that a handler returns a record without ever asking whose record it is. So on my side it splits cleanly: static layer for pattern-shaped bugs (fast, cheap, seed-tested), AI layer for absence-shaped ones like IDOR. I haven't seed-tested the AI layer yet — that's the next experiment, and it's harder, because you can't grep your way to "did it notice what wasn't asked."
Your post basically handed me the test harness. Thanks for running the messy version so the rest of us could copy the homework.
It's great to see teams proactively testing tools like Semgrep to improve their security posture. From an engineering perspective, false negatives are often more insidious than false positives in static analysis — especially when you're learning. I've seen similar patterns when evaluating tools for GPU-driven CI/CD pipelines, where subtle misconfigurations can be easy to miss. Keep up the good work!
Thanks! That was one of the biggest lessons for us too. False positives are annoying, but false negatives can give you confidence that you haven’t actually earned.
And yes, subtle CI configuration issues ended up being a bigger part of the experiment than we expected.