Before you read: we are not security experts. We are a small team learning security tooling by testing it directly and writing down what happened. This is a field note, not a best-practices guide. If something here is wrong, corrections in the comments are welcome.
TL;DR
We put 10 common vulnerabilities into a React and TypeScript app on purpose, then ran the free, zero-config Semgrep (--config=auto) against them. It reported 3 of the 10. The gap is not a defect in Semgrep. It reflects what free static analysis is built to find and what it is not. The details are below.
[IMAGE PLACEHOLDER, COVER] A cover image goes here. The scorecard graphic or a screenshot of the GitHub Actions run both work.
Why we ran this
TL;DR: We wanted a measured answer to one question. On its own, how much does a free SAST scan actually find?
"Add SAST to your CI" is common advice. The part that rarely gets measured is what you get from it when you pay nothing and write no custom rules. Instead of guessing, we set up a small test:
- A Security Operations Dashboard built with React 18, TypeScript, and Vite.
- A folder,
src/security-test-cases/, holding 10 deliberately vulnerable files, one per weakness. - A GitHub Actions workflow that runs
semgrep scan --config=autoon every push.
Then we read the CI logs and recorded the results.
How Semgrep works
TL;DR: Semgrep parses your code into a tree (an AST) and matches rule patterns against that tree. It does not run your program. If no rule describes a given pattern, that pattern is not reported.
The same idea as a text diagram:
Source code -> Parser -> AST -> Match rules -> Findings
(.ts/.tsx) (a tree of (~1074 (file + line
your code) patterns) + rule id)
A few things follow from this design:
- It is fast and cheap to run. There is no compile or execution step, so a full repo scans in seconds, which suits CI.
- It handles bad code that is present. A call like
crypto.createHash('md5')has a stable shape that a rule can match. - It struggles with correct code that is absent. The missing presence of an authorization check is hard to express as a pattern.
The 10 cases we planted
TL;DR: One file per weakness, each mapped to its CWE identifier. All are isolated and never imported by real application code.
| # | Vulnerability | CWE | Risk in one line |
|---|---|---|---|
| 1 | Hardcoded credential | CWE-798 | Secrets in git history are hard to rotate and are exposed to anyone with repo access |
| 2 | IDOR | CWE-639 | Trusting a user-supplied ID lets one user read another user's data |
| 3 | Sensitive info leak (logs or localStorage) | CWE-532/312 | Tokens in logs or localStorage are easy to read |
| 4 | Broken crypto (MD5) | CWE-327 | MD5 is unsafe for password or integrity use |
| 5 | XSS via dangerouslySetInnerHTML
|
CWE-79 | Rendering raw user HTML lets a script run in another user's session |
| 6 | SQL or NoSQL injection | CWE-89 | String-concatenated queries let an attacker rewrite the query |
| 7 | Insecure deserialization or unsafe eval
|
CWE-502 | Evaluating untrusted input allows code execution |
| 8 | CSRF (no token) | CWE-352 | Cookie-only state changes can be triggered by another site |
| 9 | Open redirect | CWE-601 | Redirecting to an unchecked URL enables convincing phishing |
| 10 | Improper error handling | CWE-209 | Raw stack traces reveal internal paths and logic |
The setup did not work on the first try
TL;DR: The pipeline broke several times before it produced results. Each failure had a specific cause. The sequence is below.
Attempt 1 Failed. We used a flag that does not exist, `--soft-fail`.
CI stopped with "unknown option". Lesson: check the CLI reference, not memory.
Attempt 2 Failed. The SARIF upload hit "Resource not accessible by integration".
Cause: missing `permissions: security-events: write`. Tokens are read-only by default.
Attempt 3 Partial. On a private repo, GitHub code scanning (Advanced Security) was unavailable.
We made the upload non-blocking. The Security tab is not free on private repos.
Attempt 4 Not useful. `--sarif --output=file` suppressed the readable report.
The log showed only a count, no paths. We split it into two steps.
Attempt 5 Worked. Findings appeared in the log with path, line, and rule id.
Most of our time went into configuration rather than security: flags, permissions, and output formats. That is worth planning for.
Results
TL;DR: The first round, with generic mock code, reported 1 of 10. After we rewrote the fixtures to resemble real framework code, it reported 3 of 10. The remaining 7 stayed unreported across six different rule packs.
| # | Vulnerability | Round 1 (generic) | Round 2 (realistic) |
|---|---|---|---|
| 4 | Broken crypto (MD5) | ✅ | ✅ |
| 7 | Unsafe eval()
|
❌ | ✅ |
| 9 | Open redirect | ❌ | ✅ |
| 1 | Hardcoded secret | ❌ | ❌ |
| 2 | IDOR | ❌ | ❌ |
| 3 | Sensitive info leak | ❌ | ❌ |
| 5 | XSS | ❌ | ❌ |
| 6 | SQL or NoSQL injection | ❌ | ❌ |
| 8 | CSRF | ❌ | ❌ |
| 10 | Stack-trace leak | ❌ | ❌ |
The unplanned finding
TL;DR: The same scan reported 3 real issues we did not plant, which is useful evidence that the tool works on real code.
- A mutable GitHub Actions tag (
@v4rather than a pinned SHA), which is a supply-chain risk. - An incomplete-sanitization issue in our actual
metricsApi.ts. - A ReDoS risk from a non-literal RegExp in our actual
emlParser.ts.
The 7 misses are not a sign of a broken scanner. On production code, it reported genuine problems.
Why 7 were not reported
TL;DR: Three structural reasons, none of them a bug. Pattern matching has limits.
It matches shapes, not runtime behavior
A static matcher cannot follow a value from an HTTP request through several functions to a dangerous call unless a rule already describes that flow. Deeper cross-file taint tracking is mostly a paid feature.
Missing-check bugs are hard for generic rules
IDOR and CSRF are not cases of bad code being present. They are cases of expected code being absent, such as a missing ownership check or a missing CSRF token. A generic community rule has no way to know what your application's auth is supposed to look like.
Secrets detection is a separate tool
Finding hardcoded keys relies on entropy analysis and provider-specific patterns, closer to TruffleHog or GitGuardian than to AST matching. Our AWS and Azure style placeholders were not reported even by the dedicated secrets pack, because that capability is a different product.
What static SAST handles well, and what it does not
| Where static SAST does well | Where it does not |
|---|---|
Known dangerous functions such as eval and md5
|
Business-logic flaws such as IDOR and broken access control |
| Risky configuration such as mutable CI tags | Cases where a required check is simply missing |
| Syntactic anti-patterns | Data flow across multiple files, without paid taint tracking |
| Fast, cheap, consistent at scale | Secrets, which need a dedicated scanner |
Is it still worth running? Yes, as a first layer
TL;DR: A clean Semgrep run means no known pattern matched. It does not mean there are no vulnerabilities. Use it as the fast first filter, not the only control.
What we would suggest for a small team:
- Keep Semgrep OSS in CI. It is free and reported 3 real issues for us, so there is no reason to remove it.
- Do not read "0 findings" as "secure". It means no generic pattern matched.
- Write a small number of custom rules against your own auth helpers, for example flagging any
/api/handler that does not callrequireOwnership(). A handful of targeted rules will find more of your IDOR and CSRF cases than any generic pack. - Add a dedicated secrets scanner. Do not rely on
--config=autofor keys. - Keep code review in the loop. A reviewer asking whether an endpoint checks ownership catches what a free static tool will not.
What we took away from it
TL;DR: The value of a tool comes from knowing where it stops.
- A green check is a starting point, not a guarantee.
- Much of the work is configuration, not analysis. Plan for that.
- Free SAST, custom rules, a secrets scanner, and code review work together. No single tool covers all of it.
- Measuring your own coverage, such as this 3 of 10, is more useful than a vendor's claim.
If you are early in learning security, a small test like this is worth building. Testing the tool directly taught us more in a day than reading about it would have.
We are still learning, so corrections and additions in the comments are welcome.
Top comments (0)