DEV Community: Zarif Latif

A pentest report is the start of the work. Here is what the end should look like.

Zarif Latif — Fri, 16 Oct 2026 05:57:55 +0000

A pentest report lists what is wrong. What you show an auditor, or an enterprise customer's security team, is the opposite: what was wrong, what changed, and the proof it worked. Teams usually have the first document and have to build the second in a hurry.

This is a practical list of what that second document should hold, taken from a real exercise we ran on OWASP NodeGoat. We are not auditors, and every auditor and customer asks for slightly different things, so check yours. But the pattern below is what most reviewers want to be able to follow.

Six things to keep for every finding

For each finding in the report, someone reading your records later should be able to find:

What it was. A stable ID, the weakness (for example CWE-601, open redirect), and the file and line.
That it was in scope. The list of findings you agreed to close, fixed in writing before the work started, so nothing is added or quietly dropped.
What changed. A pull request, one per weakness, whose description says what changed and why.
Who made and reviewed the change. The author, and your own team's merge.
That it worked. A rescan after the fix showing the finding is gone and nothing new appeared.
When. The merge date, so the record shows the finding was closed inside the audit period.

A spreadsheet with those six columns, one row per finding, is most of what a reviewer asks for. The pull requests and the rescan output are the proof behind each row.

A real row, and the one we did not change

On 14 October we ran a rehearsal Fix Sprint on OWASP NodeGoat, a deliberately vulnerable Node.js app. Sixteen findings were in scope. Fifteen were fixed in five pull requests: six by Railo's rule-based fixes, nine written by hand by an engineer. One was explained instead of changed. The evidence pack tells you which is which.

Here is one row, as it appears in the pack:

ID: F-004
Weakness: open redirect (CWE-601)
Where: app/routes/index.js, line 72
Fixed by: Railo
Check: finding gone, nothing new
Pull request: 1
Merged: 2026-10-14

The finding that was not fixed is the more interesting one. F-007 is a TLS private key committed to the repository. The tempting fix is to delete the file. That does not help: the key stays in git history and has to be treated as exposed. So the pack records it as "explained, not changed", with the real recommendation: create a new key and certificate, keep them outside the repository, remove the file, and rewrite history if the repository is or was ever public.

A final scan of the merged code found only that one finding. An auditor can read that honestly: everything was closed except one item, and the reason it is still listed is written down.

The full pack is public, with the CSV: railo.dev/sample-evidence.

Three ways closing a finding goes wrong

Closing it by deleting the evidence. Removing a file, a line or a scanner rule makes the finding disappear without fixing anything. Records should show a change that removes the weakness, checked by a rescan.
Merging without a rescan. A fix can remove one finding and introduce another, or break the build. The rescan is what shows the finding is gone and nothing new appeared, and your own CI on the change is still the real test.
One giant pull request. Forty fixes in a single change are hard to review and impossible to trace. One pull request per weakness lets a reviewer read each fix, and lets the record point from each finding to its own change.

How a Fix Sprint produces this

We built our Fix Sprint around exactly this record. You send the findings, from a pentest report, a scanner export or a customer questionnaire, and we agree in writing which ones are in scope. In one week we return a pull request per weakness. Railo's rule-based fixes cover the common patterns; an engineer writes the rest by hand and reads where the code is used before changing it. Every fix is rescanned. Your team reviews and merges, and we finish with the evidence pack: one row per finding, as above.

The price is fixed at $1,500 for one repository. Nothing is merged without your review, and nothing from your repository is executed on our side.

If you have a report to close, email zarif.latif@railo.dev with the subject "Railo Fix Sprint", or book 15 minutes at cal.com/railo.

Zarif Latif, founder, Railo

How We Tested Railo on 11 Public Repositories

Zarif Latif — Fri, 16 Oct 2026 05:54:21 +0000

Railo is a GitHub Action that finds security issues in a pull request and proposes the fix. The fixes are rule-based code transforms, not model output: the same input always gives the same patch, and every patch is a diff you can read.

Until this month we measured fix quality on a benchmark of examples we wrote ourselves. That number looked good, and it was also the easiest test we could have chosen. So we ran the exact pipeline the Action runs on eleven public repositories we did not write, and read every fix it proposed.

The short version:

923 findings across the eleven repositories.
162 proposed fixes. We read all of them.
95 of those 162 should not have been proposed. Most were edits to vendored libraries, test data and seed data.
None of the 162 broke the syntax of a file. Wrong is not the same as broken, and wrong was the problem.

This post is the list of what went wrong and what we changed. We think it is more useful than another benchmark score.

How we ran it

We picked two kinds of repository on purpose:

Deliberately vulnerable apps, where real findings are guaranteed: OWASP NodeGoat, OWASP Juice Shop, PyGoat, DVWA, OWASP WebGoat and govwa.
Well-maintained projects, where almost every finding should be noise: Flask, Express, Gin, Slim and Spring Petclinic.

For each one:

Shallow clone, and scan the whole repository rather than a pull request diff.
Run the Action's own code path: load the config, scan with the pinned Semgrep version (1.102.0), generate fixes, collect the changed files.
Parse every changed file before and after the fix with the language's own tooling: Python's compile, the TypeScript parser for JavaScript and TypeScript, gofmt -e, php -l, and javac stopped after parsing. A fix that adds a parse error counts as breaking the file.
Read every proposed fix by hand.

One thing we switched off: the Action normally tries the project's own test command before and after each fix. In this harness we did not execute anything from the repositories.

We now re-run this harness, and read the diffs, before every Action release.

Results

Two things stand out: almost all the wrong fixes landed in code the project did not write or does not ship, and on the well-maintained projects the right number of fixes was zero.

Findings, then fixes proposed before and after our changes:

WebGoat: 381 findings, 95 fixes before, 15 after
Juice Shop: 216 findings, 11 before, 6 after
PyGoat: 132 findings, 16 before, 16 after
DVWA: 94 findings, 27 before, 24 after
NodeGoat: 24 findings, 10 before, 6 after
govwa: 8 findings, 0 before, 0 after
Flask: 22 findings, 1 before, 0 after
Gin: 20 findings, 1 before, 0 after
Express: 14 findings, 0 before, 0 after
Spring Petclinic: 9 findings, 1 before, 0 after
Slim: 3 findings, 0 before, 0 after
Total: 923 findings, 162 fixes before, 67 after

WebGoat alone accounts for 80 of the 95 removed fixes, and about 77 of those were in two vendored editor files.

A follow-up release then cut noise on the maintained projects: Flask went from 22 findings to 11, and NodeGoat from 24 to 21, with one more correct fix (7 instead of 6).

What we got wrong

Every problem below was a fix that parsed, looked plausible in isolation, and was wrong in context. Each case now goes to a person instead: the finding is still reported, but no change is proposed. Each has a regression test.

1. Rewriting vendored libraries

WebGoat ships the Ace editor in static/js/libs/. Our hard-coded-secret rule saw lines like token: "keyword" in Ace's syntax-highlighting rules and turned them into process.env.TOKEN. About 77 fixes, all nonsense. Juice Shop's vendored three.js was edited too.

Now: lib/ and libs/ under static asset folders, minified and bundled files, and any JavaScript file over 5,000 lines are treated as third-party code.

2. Treating test data as production

Gin keeps a certificate and key in testdata/ for its TLS tests. We "fixed" them.

Now: testdata, __tests__, __mocks__, mocks, e2e, cypress, bower_components and similar folders are skipped.

3. Collapsing seed data into one variable

NodeGoat's db-reset.js creates three users with different passwords. We gave all three the same process.env.PASSWORD. Juice Shop's users.yml was worse: 27 users, all set to the literal string ${PASSWORD}.

Now: several different values for the same key in one file mean data, not a secret.

4. Writing `${VAR}` where nothing substitutes it

We rewrote a value in Petclinic's Kubernetes manifest, and in Juice Shop's YAML data, to ${VAR}. Neither file is ever processed by anything that expands it, so the app would have read the literal text.

Now: only Docker Compose files and Spring Boot config files get ${VAR}, because those are the formats that substitute it.

5. `process.env` in browser code

Angular components and in-browser lesson scripts got process.env.SECRET. There is no process.env in a browser.

Now: browser code, detected by path, file extension, or use of document, window or localStorage outside string literals, is not rewritten. (The first version of this check mistook a server config file that mentioned document.write inside a string for browser code. Fixed in the follow-up release.)

6. Names that only look like secrets

A smart-contract address called BeeTokenAddress. DVWA's $token = "userid:2", which is an attack payload, not a credential. Ace's token types again.

Now: names ending in address, id, url, host, name, type and similar are left alone, as are token values that are not shaped like a credential, unless the value matches a known provider format (AWS, GitHub, Slack, Stripe, Google keys), which is caught anywhere.

7. `innerHTML` to `textContent` on HTML that is meant to be HTML

Juice Shop renders Markdown to HTML with snarkdown and assigns it to innerHTML; WebGoat builds tables as HTML strings. Switching to textContent would print the tags as text.

Now: values that are HTML by construction are not rewritten.

8. Flask `Markup(value)` to `escape(value)`

This double-escapes markup that Flask already treats as safe.

Now: Markup() with no request data in it is not rewritten.

9. "Escaping" a JSONP callback with `htmlspecialchars`

It makes the code look handled without making the callback safe.

Now: this goes to a person.

What is still not good enough

Noise on well-maintained code. After the follow-up release Flask still produces 11 findings, all of which we read as noise: 7 from a generic open-redirect rule, 2 from a compile() rule, one Markup and one SHA-1. No fix is proposed for any of them, but in warn mode they still show up on Flask's pull requests. That's the next thing we are working on.
Partial detection. In DVWA's view_help.php, the if branch is flagged and fixed and the identical else branch is not.
Context a rule cannot see. Making Juice Shop's token cookie httpOnly is correct in general and breaks Juice Shop, whose frontend reads that cookie. We offer it as a suggestion for review, not an automatic change, which is where it belongs.
Not automated, by design. NodeGoat's eval(req.body.preTax) is detected and explained, but a person writes the fix. Some fixes need a person, and we would rather say so than guess.

If you maintain one of these projects and think we have something wrong, we would like to hear it.

Try it

The Action is free to use and needs no account or key. It covers Python, JavaScript and TypeScript, Java, Go and PHP. Start in warn mode: it comments the proposed fixes on the pull request and changes nothing.

- uses: actions/checkout@v4
  with:
    ref: ${{ github.head_ref }}
    fetch-depth: 0
- uses: IWEBai/railo-action@v2
  with:
    mode: warn
    base_branch: ${{ github.base_ref }}
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

The full workflow, with the permissions it needs, is on the Action's page. If you run it on your own code and it proposes something wrong, please open an issue with the diff: that is exactly the kind of case this post is made of.

If you would rather not fix it yourself

The Action covers the common, pattern-based findings. Plenty of real findings are not that: a pentest report with forty items, a scanner export before an audit, a customer's security questionnaire. For those we run a Fix Sprint: you send us the findings, and in one week you get a pull request per weakness. Some of the fixes Railo writes; the rest an engineer writes by hand, and the evidence pack says which. Every fix is rescanned, and your team reviews and merges everything. It is a fixed price of $1,500 for one repository.

The result is an evidence pack: one row per finding, with the file, the fix, who made it, the rescan result and the merge date. We ran a rehearsal on OWASP NodeGoat, and the finished pack is public: railo.dev/sample-evidence.

If you have findings to close, email zarif.latif@railo.dev with the subject "Railo Fix Sprint", or book 15 minutes at cal.com/railo.