DEV Community: omarmohelal

I pointed my own security tool at my own GitHub Action. It found two bugs.

omarmohelal — Wed, 09 Sep 2026 12:35:51 +0000

I write an application-security tool called SecHelix. Last week I added a GitHub
Action to it. Before merging, I pointed the tool at its own new Action.

It found two real defects. Neither would have failed a test. Both were the kind
of thing I would have shipped.

This is a post about those two bugs, because they are better arguments for
evidence-first review than anything I could write about the methodology.

Bug one: the artifact was the wrong copy

The Action runs an audit and uploads the result as a build artifact so you can
download it from the workflow run. The step looked like this:

- run: sechelix audit "$PATH" --json > sechelix-run.json
- uses: actions/upload-artifact@...
  with:
    path: sechelix-run.json

Obvious. Works. Ships.

Here is what I had forgotten about my own codebase. The runner writes two copies
of every result. The one it persists to disk goes through storage.write_json,
which runs the payload through a redactor. The one --json prints to stdout is
result.to_dict(), raw.

I only had to run the redactor against a payload to see it:

persisted (storage.write_json):  "authorization": "[REDACTED]"
stdout    (audit --json):        "authorization": "Bearer sk-live-abc123"

So the Action was uploading the unredacted projection into a build artifact that
anyone with read access to the repository can download, for the seven days it
lives. During a security review, of all things — the exact run most likely to
have a secret quoted in a node payload, because that is what it was looking for.

The fix is one line: re-read the persisted copy with
sechelix report --format json before uploading anything.

The interesting part is not the fix. It is that there were two projections of
the same object with different safety properties, and only one of them was
documented as redacted. No test could have caught this, because both
projections were behaving exactly as written. The defect was in the seam.

Bug two: an output value could forge another output

GitHub Actions steps communicate through a file:

echo "outcome=BLOCKED" >> "$GITHUB_OUTPUT"

It is newline-delimited. So a value containing a newline writes a second key.
And when a key appears twice, the runner takes the last one.

Which means a value like this:

RUN-X
outcome=PASS
blocking-count=0

produces:

outcome=BLOCKED
reason=1 verified finding(s) at CRITICAL or HIGH severity are open.
run-id=RUN-X
outcome=PASS          <-- forged
blocking-count=0      <-- forged
incomplete=false
blocking-count=1

outcome=PASS wins. A gate that said BLOCKED reports PASS.

Now — the honest version. In this Action's actual flow, run_id is generated by
the runner and constrained by a regex that cannot contain a newline. This was
not a reachable bypass. I have seen this class of thing written up as critical
by tools that never checked reachability, and it is exactly the kind of finding
that costs a reviewer an afternoon and costs the tool its credibility.

So it is reported at its real severity: defence in depth, fixed because the fix
is four lines and it makes the property hold for every field instead of for
one field by accident.

I wrote the regression test first and confirmed it red:

AssertionError: 7 != 5 : ['outcome=BLOCKED', ..., 'outcome=PASS', ...]

Why this is the whole argument

Most security tooling optimises for finding more. The expensive failure in
AppSec is the opposite: a queue of confident findings where a third are wrong,
no cheap way to tell which third, and after the second bad one nobody reads the
third.

So SecHelix is built the other way round. Every candidate goes to an independent
verifier whose job is to disprove it, and the report includes what it refuted
and why. A run that could not analyse anything returns INCOMPLETE, never a
clean PASS.

Both bugs above are that idea working. Bug one survived refutation and got
reported at Medium with a reproduction. Bug two survived as a real defect but
failed the reachability test, and got reported as defence in depth instead of
as a critical gate bypass. The second outcome is the one I care about, because
anyone can build a tool that shouts.

Try the version that takes 90 seconds

There is a demo in the repo that is small enough to read in one sitting. Two
candidates in a small multi-tenant expense API:

git clone https://github.com/omarmohelal/SecHelix && cd SecHelix
python examples/expense-api/prove.py

Python 3.10+, no dependencies, no network, no containers.

The one every pattern matcher flags builds a SQL ORDER BY with an f-string.
It is not exploitable. The sort key is mapped onto one of five module constants,
and the proof checks by identity that the object reaching the query text is
that constant — so it is not "the payloads I tried were rejected", it is "no
string derived from the request can survive the mapping".

The one scanners walk past is three lines below. The endpoint has an
authorization check. It is correct. An employee token really does get a 403. And
it asks "may someone like you open a receipt?" instead of "is this receipt
yours?", over a query scoped by nothing — so any approver at any tenant reads any
receipt by id.

That is the difference in one screen: a check that is present and answers the
wrong question is invisible to a tool looking for a missing check.

The demo goes through the whole loop — reproduce, root cause, a two-line fix,
and a regression test that was red before the fix. CI drives it in all three
states on every run (vulnerable, patched, reverted), so the README cannot drift
into describing something that no longer happens.

What I am not claiming

There is no detection-rate number, and there is not going to be one from me.
The public benchmark position is NOT_MEASURED, deliberately — I am not going to
grade my own homework and publish the score.

If you want to score it, or anything else, there is a separate
challenge repo: ten cases,
three of them decoys, a false positive costs exactly what a miss costs, UNKNOWN
scores zero rather than counting as an error, and the ground truth is public and
arguable. It is deliberately tool-neutral. If it makes SecHelix look bad, that is
information too.

SecHelix is Apache-2.0 and works as an Agent Skill with no runtime required:

npx skills@latest add omarmohelal/SecHelix --skill sechelix

If it is useful in your security workflow, a star on
the repository helps other people
find it.

Why my security scanner refuses to tell you your code is clean

omarmohelal — Sat, 05 Sep 2026 10:18:40 +0000

There is a failure mode in security tooling that almost nobody instruments for,
and once you see it you cannot unsee it.

An empty findings list looks identical to a clean result.

A scanner that crashed halfway, a scanner whose API key expired, a scanner that
ran out of budget, and a scanner that genuinely examined everything and found
nothing — all four produce the same artifact: findings: []. The CI job goes
green. The PR merges. Nobody involved can tell which of the four happened.

I built an application-security agent, and this is the problem I ended up
designing the whole thing around.

What it does instead

pip install sechelix
sechelix audit examples/demo-app

  BLOCKED    authorization   no reasoning executor configured; this node
                             analyses code and cannot be answered by the
                             runner alone
  SUCCEEDED  map
  BLOCKED    verify          dependency not satisfied: authorization
  BLOCKED    gate            dependency not satisfied: verify

RESULT  INCOMPLETE - unsatisfied mandatory nodes: gate, verify
        No security claim can be made from this run.

Exit code 1. Not because your code is bad — because nothing was examined,
and the tool will not pretend otherwise.

The states are deliberately distinct:

State	Meaning
`SUCCEEDED`	the lane ran and delivered
`SKIPPED`	the lane does not apply to this target — it owes no evidence
`BLOCKED`	the lane could not run. A real question is open
`FAILED`	the lane ran and errored

SKIPPED and BLOCKED are the pair that matters. An inapplicable lane is a
real answer. An unaffordable verifier is not.

The budget case is the one that convinced me

Give a run a cost ceiling and let it run out just before the verifier:

map     SUCCEEDED
authz   SUCCEEDED
verify  BLOCKED    max_cost_usd budget exhausted: requested 0.5, 0.1 remaining
gate    BLOCKED    dependency not satisfied: verify

unsatisfied mandatory: ['gate', 'verify']

A budget limit that silently skips verification and then returns PASS has
converted a cost control into a correctness bug. That is strictly worse than
having no budget at all, because now the failure is invisible. There is a test
that starves the budget mid-verification and asserts the gate never passes.

Blinding the verifier

The second design decision: a hunter proposes findings, and an independent
verifier tries to refute them. The verifier is not told the hunter's
confidence, severity, verdict, or exploitability — those fields are stripped
before the candidate is handed over.

The reason is mundane. A verifier that reads "HIGH confidence SQL injection,
definitely exploitable" before it looks at the code is not verifying. It is
agreeing.

I tested this by planting a fake finding. Two candidates went in: one real
(against a file with a genuine missing ownership check), one fabricated against
that file's clean twin, tagged as loudly as I could make it —
confidence: HIGH, severity: CRITICAL, verdict: exploitable,
hunter_notes: certain this is a real IDOR.

The verifier kept the real one and refuted the plant:

refuted: the candidate locating the missing-ownership claim at
clean/orders.py:get_order is false. The clean variant explicitly compares
order['user_id'] to session['user_id'] and returns (None, 404) on
mismatch before the success return.

It named the exact line. And a capture of the prompt actually sent confirms
HIGH, CRITICAL, exploitable and the hunter's note never reached it.

Paired fixtures, because precision is the hard part

The demo corpus has five vulnerable files and five clean counterparts, each pair
differing in exactly one security-relevant way.

This is not decoration. A corpus that only contains bugs cannot measure
whether a tool stops. An agent that flags every line scores perfectly on a
vulnerable-only benchmark and is useless in a real repository. If the clean twin
is not in the corpus, a false positive is invisible.

It found a bug I wrote

I pointed the tool at its own runner. It found a path traversal:

run_id arrives from the command line — sechelix report <id> — and was joined
straight onto the runs directory. So:

run_id='../../outside.json'  ->  resolves outside the workspace

report would then read that file and print it. Fixed with a shape check plus a
resolved-path confinement check, and ten regression tests. It is in the history
at commit 0e50b56 rather than quietly patched.

What I have not measured

There is a blind evaluation: 76 cases judged by 76 independent processes, each
launched from an empty directory containing only the case file — precision
0.950, false-positive rejection 0.947. That is a label-only result on 38
authored pairs. It is not a measurement of the full workflow, which is still
marked NOT_MEASURED in the repository.

I also have not benchmarked against SEC-AF, Cloudflare's security-audit skill,
or Strix. So I am not claiming to beat them. The comparison does not exist yet,
and saying so is cheaper than being caught.

Try it

# Agent Skill — Claude Code, Codex, Copilot. No Python.
npx skills@latest add omarmohelal/SecHelix --skill sechelix

# Optional runner
pip install sechelix
sechelix doctor
sechelix audit examples/demo-app

STATIC is the default and performs no network access at all. No account, no
email, no cloud.

Apache-2.0 · github.com/omarmohelal/SecHelix

If you disagree with the fail-closed design — and there is a real argument that
a noisy INCOMPLETE trains people to ignore it — I would genuinely like to hear
it.

Why an AppSec agent should try to disprove its own findings

omarmohelal — Wed, 02 Sep 2026 12:49:43 +0000

Originally published on sechelix.com. This is a syndicated copy; the canonical version lives there.

The failure mode nobody optimises for

Point a capable model at a codebase and ask it to find security problems, and it will find some. It will also produce findings that are confident, well-structured, and wrong — and those read exactly like the real ones, because nothing about a fluent explanation requires the underlying claim to be true.

That asymmetry is the actual problem. A missed bug costs you the bug. A plausible false positive costs a reviewer an hour, and the third one costs you their attention for every finding after it.

Teams do not abandon bad security tooling because it misses things. They abandon it because they stop believing it. Once a tool has been wrong three times in a way that felt authoritative, its next report gets skimmed — including the one that was right.

Most effort in this space goes into detection. I think the more useful lever is refusal.

A finding is a claim

The design premise of SecHelix is one sentence: a security finding is a claim, and a claim gets an independent refutation attempt before anyone is told about it.

Three mechanisms carry most of that.

Applicability has four outcomes, and one of them cannot be laundered

Every check resolves to APPLICABLE, NOT_APPLICABLE, UNKNOWN, or BLOCKED. UNKNOWN and BLOCKED can never be converted into NOT_APPLICABLE.

This sounds like bookkeeping. It is the single change with the largest effect on what a report means.

Without it, "we could not evaluate this" and "this does not apply here" collapse into the same clean-looking output, and a reader cannot tell a check that passed from a check that never ran. With it, a clean report becomes a much stronger statement, because everything the tool could not establish is still visible on the page.

The verifier's job is to lose

Every candidate goes to an independent verifier prompted to disprove it — not to double-check it, to attack it. Attacker control. Reachability. Missing guard assumptions. Role preconditions. Whether the vulnerable state is producible at all. Whether some compensating control already blocks the exploit.

Findings that survive carry a seven-link evidence chain, and a finding that cannot name its links does not ship. High and Critical additionally require regression proof: the assertion has to fail against the vulnerable control and pass after the fix, or the claim that it was fixed is itself unverified.

The gate fails closed

The release decision is PASS, PASS_WITH_KNOWN_RISK, BLOCKED, or INCOMPLETE. Missing required evidence returns INCOMPLETE and a non-zero exit — never a silent pass.

A gate that passes when it cannot see is worse than no gate, because it manufactures confidence.

What this looks like on real code

An authorized self-audit of a small Next.js storefront I own. STATIC and LOCAL mode, zero scanners enabled, nothing outside 127.0.0.1 contacted. One external data source, forty-one of 546 hypotheses applicable, three candidates.

One verified. No CSP, X-Frame-Options or HSTS on any route. A probe page served from a separate origin framed the entire interface, including the sign-in entry point. The fix was a catch-all headers rule; the proof was the browser's own refusal on retest:

Framing 'http://localhost:3009/' violates the following Content Security Policy directive: "frame-ancestors 'none'". The request has been blocked.

It was held at MEDIUM, not High. The realistic outcome is phishing amplification, and the app performs no authenticated state-changing actions. Severity you can defend under questioning is worth more than severity that looks impressive in a summary.

One refuted. Remote configuration values reached href and src with only .trim() applied. That is precisely the shape a scanner — or a confident reviewer — reports as high-severity XSS.

Verification killed it. React 19 rewrote the payload to:

href="javascript:throw new Error('React has blocked a javascript: URL as a security precaution.')"

and attacker control over the configuration source was never established. Recorded FALSE_POSITIVE, with the refutation reasoning kept rather than deleted.

A scheme allowlist was added anyway — and labelled hardening, not a vulnerability fix, because calling it a fix would imply there had been a vulnerability.

That second result is the one I would point at. Any tool can produce findings. Discarding one is the part that costs something.

And one process finding, recorded because it is the interesting kind of near-miss. The first retest appeared to fail. A stale prerender cache and a server still bound to the old port made a real fix look broken — which is exactly the mechanism by which a real fix silently becomes a false claim of remediation in the other direction. Only a clean rebuild settled it.

The same discipline, turned inward

The uncomfortable version of this argument is what happens when you point the tool at itself.

Running the differential reviewer over its own changes produced findings for a docstring explaining that a digest "is not a signature" — classified as a webhook change — and a comment about a sample-size bucket, classified as storage access. Neither line does anything. The tool was flagging its own prose.

Writing tests that tried to break the fail-closed guarantees, rather than confirm them, found worse. An unresolved CRITICAL candidate passed the release gate, because the status enum covering "unproven" listed several states and not the most common one. An empty commit string made every report read as current, because two revisions were compared on the shorter of their lengths and nothing enforced a floor — and an empty string is exactly what git rev-parse returns when it produces no output rather than failing.

Both were fail-open paths in modules whose stated contract is fail-closed. Both are fixed. Both are in the changelog under their own heading, because a project arguing that findings need proof does not get to be quiet about its own.

What I am not claiming

The benchmark was unpublished for months, on purpose. The repository contains 38 paired vulnerable/clean fixtures and a working scoring harness. It also contained a machine-readable blocker, CONTAMINATED_EVALUATOR, recording why no number was published: the fixtures were authored by assistant sessions working in the repository, so scoring one of those sessions measures recall of answers it wrote rather than security-review capability.

A sealed blind packet exists so an uncontaminated evaluator can produce a real measurement. On 2026-09-02 one finally ran: 76 cases, each judged by a separate process that had never seen the repository, the fixtures, the labels or the pairings. The committed result is evals/results/claude-sonnet-5-blind-2026-09-02.json.

Read that result narrowly. It measures a single-pass, label-only judgment — one question per file, one label back. There was no attack-surface pass, no independent refutation pass, no adapters, no evidence chain and no release gate. It is not a measurement of the workflow this article describes, and applicability_accuracy, regression_proof_rate and release_gate_accuracy are still the literal string NOT_MEASURED. One model, one run, on an authored and balanced suite. No comparison to any other tool is offered or implied.

Everything else on the list still stands:

The committed keyword baseline is not a score. It is a regex matcher, flagged is_sechelix_result: false, that lands at chance on the fixture suite. It exists to prove the scoring harness works and that the fixtures resist pattern matching.
One case study is not evidence of general performance. One small app, self-audited by the author, one verified MEDIUM finding, one refutation.
It is alpha. Contracts and interfaces can still change.
It is built with substantial AI assistance, directed by a human. Saying otherwise would be its own kind of unverified claim.

Try it

npx skills@latest add omarmohelal/SecHelix --skill sechelix

Then ask your agent for an authorized audit of a repository you own.

Apache-2.0, Python standard library only: github.com/omarmohelal/SecHelix

The critique I actually want is whether a refutation step run by the same class of model that generated the candidate is meaningfully independent, or whether it only catches the shallowest errors. I have a design opinion about that — the quorum mechanism exists because I am not sure — and the label-only run above does not answer it. I would rather say so than imply otherwise.