DEV Community: Oracle Mode

JANISSARY 7.3.0: F1 0.921, reproducible in one command

Oracle Mode — Sun, 25 Oct 2026 01:55:26 +0000

JANISSARY 7.3.0: F1 0.921, reproducible in one command

7.3.0 is out. Code freeze — functionally complete, frozen to prevent regressions. Release notes: github.com/Hetaireia/janissary/releases/tag/7.3.0. Docker: oraclemode337/janissary:7.3.0. PyPI: janissary==7.3.0.

TL;DR: F1 0.921 on the OWASP Benchmark DAST-applicable corpus. Precision 1.000. Committed run artifacts. One command to reproduce. And here's what it misses, before you install.

The benchmark number — reproduced, not asserted

The 0.920 figure from the earlier post was measured on v7.2.0. The v7.2.0 run artifacts weren't committed, so it was project-reported, not reproducible. That was the honest but weak position.

This is the strong one:

DAST-APPLICABLE  TP 987  FP 0  FN 169  TN 1102  P 1.000  R 0.854  F1 0.921

Reproduced on v7.3.0 against the same pinned target (OWASP Benchmark commit 20cbf3d111…), under the pre-registered methodology in BENCHMARK.md. Amendment 21 records the protocol, the deviation from the 3-run baseline (documented, with reasoning), and the exact reproduce command.

Two numbers, both traceable:

Single-pass baseline: F1 0.902 (TP 950 / FN 206)
FN-recovery at JANISSARY_TIMEOUT=180: F1 0.921 (TP 987 / FN 169)

The 0.902 to 0.921 jump is the Amendment 20 recovery protocol: re-scan the misses at a longer timeout, merge, re-score. Documented in the pre-registration before the run, not after.

Delta vs Amendment 20 (v7.2.0: F1 0.920, TP 984): +3 TP, +0.001 F1 — within the ±13 TP run-to-run noise floor stated in BENCHMARK.md line 654. The claim reproduces.

The receipt: all three run directories committed under benchmark/track_a/runs/. To verify:

python benchmark/track_a/score_dast_aware.py benchmark/track_a/runs/20261024T130735Z-janissary-r1+rec

That's it. You should get TP 987 / FN 169 / F1 0.921. If you don't, the claim is wrong and you should say so.

What it's good at

Deduplication. DVWA run: 8 CRITICAL signals grouped into 1 finding. That's the thing triage-fatigued users actually need.
WAF evidence. Live ModSecurity: pacer backed off to 30.0s across 38 events. Observed backoff behavior, not a claimed bypass.
Framework-specific SQLi. Paren-closing payloads verified live.

What it misses — read before installing

SPA catch-all false positives. 39 FPs on a Node SPA in the Juice Shop run. WordPress-admin-module rule fires on catch-all routing. Issue #1.
401 aborts the scan for that path. Auth-gated targets can hide findings downstream. Issue #2.
Pathtraver recall 0.049–0.308. Not a weak result — an unsupported class. Do not use JANISSARY for path traversal in 7.x. Use something else.
One case still times out at 180s (BenchmarkTest02321, xss-04, param_name). Counted as FN. Documented in Amendment 21.

What's next

8.x gets:

Issue #4 — generalize the six-oracle pattern to close the misattribution class.
Issue #5 — --mode ci, fast and unpaced for trusted pipelines.
Payload-set generalization for framework-wrapped queries.

7.x gets hotfixes only.

Install

docker pull oraclemode337/janissary:7.3.0
pip install janissary==7.3.0

443 tests passing. main is clean at the tag.

If 7.3.0 gets something wrong, open an issue. The ledger is public.

https://dev.to/new

Oracle Mode — Mon, 19 Oct 2026 20:21:04 +0000

Why I built a differential DAST scanner instead of using Nuclei or Burp

When I started building JANISSARY, I didn't want to build another directory fuzzer or a script that blindly throws payloads at a wall. I wanted to build something that respects the operator's time.

Most automated DAST tools fall into two camps: they either generate massive amounts of noise (high false positives), or they get instantly IP-banned by WAFs the moment they start scanning.

Here is why I took a different architectural approach.

1. Differential Detection (Precision = 1.000)

Nuclei is fantastic for CVE signature matching, but blind payload fuzzing drowns you in false positives. JANISSARY uses differential detection. Every finding is gated by a baseline comparison. If a single response changes, it doesn't emit a finding. It only fires when multiple oracles (status changes, length anomalies, error signatures) agree.

On the OWASP Benchmark DAST-applicable corpus (2,258 cases), this architecture scored an F1 of 0.920 with a Precision of 1.000. Zero false positives.

2. WAF-Aware Pacing

If you are scanning an authorized target, getting blocked mid-scan wastes your time. I ran JANISSARY against OWASP Juice Shop sitting behind a ModSecurity CRS WAF. The scanner detected the WAF, started hitting block signals, and dynamically backed off. It ramped the delay up to 30 seconds, recovered on a clean streak, and still successfully found the SQL injection without aborting the scan.

3. Burp-Native Output

I don't want to fight with proprietary dashboards. JANISSARY outputs findings as JSONL with raw HTTP/1.1 requests and responses (CRLF line endings). You can pipe it straight into jq or paste the raw requests directly into Burp Suite Repeater to validate the finding manually.

The Reality Check

I recently ran JANISSARY against DVWA (Damn Vulnerable Web Application). It successfully grouped 8 distinct signals into a single CRITICAL SQL injection finding.

It’s not perfect. I document the known limitations openly in the repo—like how SPAs can cause admin-panel false positives, or how 401s abort the scan. But it is honest, it is measured, and it is fast.

You can try it out via Docker or PyPI:

docker run --rm oraclemode337/janissary scan -u "http://target.example" -p q

Repo: https://github.com/Hetaireia/janissary

0.920 on the benchmark. Zero findings on Juice Shop.

Oracle Mode — Mon, 19 Oct 2026 18:02:40 +0000

Three days ago I published JANISSARY's benchmark results: F1 0.920, precision 1.000 on the OWASP Benchmark DAST-applicable subset. Zero false positives across 2,258 cases.

Today I ran it against OWASP Juice Shop. It found 39 vulnerabilities. All 39 were wrong. And it missed the SQL injection I already knew was in the search endpoint.

Both of those things are true. This post is about why.

What the benchmark actually promises

The benchmark tests detection on a corpus of Java servlets. Each servlet either is or isn't vulnerable to a specific class. The scanner gets no source access. JANISSARY scored precision 1.000 — not a single finding on the corpus that wasn't a real vulnerability.

That number is real. It's reproducible. The run is in the repo.

But a benchmark measures one thing: how the tool behaves on the corpus. It doesn't promise the tool behaves the same way on a Node app.

That distinction is the entire post.

What happened on Juice Shop

Juice Shop is a Node/Express single-page app. It's the canonical intentionally-vulnerable target — thousands of people use it to practice web exploitation.

I ran:

janissary agent http://localhost:3000 --attack-confirm

Output:

Platforms:       wordpress
Modules run:     xmlrpc, admin
Modules skipped: scan
New findings:    39

Two things are wrong before you even look at the findings. Juice Shop isn't WordPress. And the scan module — the one with the differential oracles that produced the benchmark result — never ran.

The platform fingerprinter classified a Node app as WordPress. The agent then ran WordPress modules. The injection oracles sat this one out.

The 39 findings

All 39 had the same shape:

category:        admin
finding_type:    exposed-panel
discriminator:   /wp-admin/
detail:          wordpress /wp-admin/ -> HTTP 200
response_status: 200

Juice Shop's SPA router returns HTTP 200 and index.html for any path, including paths that don't exist. The admin module checked the status code, saw 200, and reported an exposed WordPress admin panel.

On a single-page app, every path returns 200. Every path was reported.

This is exactly the class of false positive JANISSARY's differential gating is designed to prevent. It fired anyway — because the admin module doesn't go through the differential gate. It's a status-code check wearing a scanner's clothing.

The miss

Juice Shop's /rest/products/search?q= endpoint is a documented SQL injection challenge. It's the first thing anyone tries.

janissary scan -u "http://localhost:3000/rest/products/search?q=apple" -p q

Output:

Total requests:  40
Findings:        0 in 0 groups

The scan ran. The baseline was stable. The WAF probe was clean. Nothing fired.

Here's the query Juice Shop builds internally:

SELECT * FROM Products
WHERE ((name LIKE '%apple%' OR description LIKE '%apple%')
AND deletedAt IS NULL)

To inject, you have to close two parens first:

apple')) UNION SELECT ...--

JANISSARY's payload set is benchmark-shaped. It sends ', ' OR '1'='1, and similar. Those are correct for the benchmark's servlets, which do simple string concatenation. On Juice Shop, ' produces a syntax error that the app swallows, returning an empty result set. No differential delta. No finding.

The payload set doesn't know about framework-wrapped queries.

What this says about benchmark numbers

The 0.920 is not fake. It measures what it says it measures: how JANISSARY performs on a corpus of Java servlets with simple injection points.

What it doesn't measure:

Non-Java platforms
Queries wrapped in parentheses or built by ORMs
SPAs with catch-all routing
Endpoints behind authentication
Any error output that doesn't match the benchmark's signature set

Those aren't edge cases. They're most of the web.

So the honest reading of "F1 0.920" is: "this tool performs at 0.920 on a well-defined class of applications with well-defined vulnerability shapes." The class is narrower than "web applications." It's roughly "Java servlet-style apps with simple injection points."

That's still a real result. It's just not the result I implied by publishing the number without qualification.

What I'm doing about it

Three issues, all public:

#1 — admin module reports false positives on SPA catch-all routing
#3 — SQLi payload set lacks framework-specific variants (paren-closing, comment terminators)
#2 — scan aborts on HTTP 401 without reporting auth-required state
None of these are launch-killers. All of them are fixable. The fixes are specific, small, and testable:
The admin probe should verify response body shape, not just status
The payload library should include ')), --, /* */, and quote-stacking variants
401 responses should report "endpoint requires authentication" instead of aborting silently

Each fix has a reproduction case from Juice Shop. Each can be verified against the same target.

The honest version of the claim

If you're evaluating JANISSARY on the strength of the benchmark number, here's what you should actually believe:

On applications that look like the OWASP Benchmark's servlets, it performs at F1 0.920 with zero false positives. That's measured.
On applications that don't, its behavior is uncharacterized. As of today, the first live-target run produced false positives from one module and misses from another.

The tool's whole thesis is that findings should be reproducible differences, not pattern matches. That thesis has to apply to the tool's own claims. The benchmark number was a pattern match on a corpus. The Juice Shop run is the difference.

JANISSARY is open source (Apache 2.0). Repo: https://github.com/Hetaireia/janissary. Issues: https://github.com/Hetaireia/janissary/issues. Benchmark methodology: BENCHMARK.md.

We refuse to emit a finding from a single response

Oracle Mode — Tue, 13 Oct 2026 08:21:28 +0000

Every DAST tool I've used has the same failure mode, and it isn't detection.
It's trust.

You run a scan. You get 200 findings. You triage 30. You discover 22 are
noise. You stop reading findings. The tool that was supposed to save you
time now costs you time, and the next time you run it you skim.

I built JANISSARY around one rule: a finding that isn't a difference is a
guess. Here's what that means in practice, why it's harder than it sounds,
and what we gave up to ship it.

What "differential" actually requires

A finding should never be emitted from a single response. That sounds like a
one-line policy. It's actually four engineering problems stacked on top of
each other.

First, you need a baseline you can trust. Before probing, JANISSARY
sends N benign requests to the same endpoint, same parameter, same headers.
If the baseline isn't stable — if the response time varies by more than a
threshold, if the body length jumps between runs — the tool refuses to
engage the timing gates and tells you so:

[q] baseline is static - timing gates disabled

That's not a bug, it's the tool being honest. A flaky baseline produces
flaky deltas, and flaky deltas produce false positives. Better to run
fewer gates than to run wrong ones.

Second, you need a probe that produces a reproducible delta. Firing
a payload and seeing the response change isn't enough. The change has to be
consistent, tied to the payload, and not explained by normal variance.

Third, you need to distinguish a delta from a WAF block. Cloudflare
returns a 403 with a challenge page — that's not an SQL injection, that's
Cloudflare doing its job. JANISSARY's pacer watches for those signatures
and feeds them into the differential gate so a block doesn't get reported
as a vulnerability.

Fourth, you need to normalise time-varying responses. CSRF tokens,
nonces, timestamps, session IDs, dynamic ad copy — the response body
changes constantly for reasons that have nothing to do with your payload.
Comparison has to strip those out or the gate fires on everything.

Each of those is a separate subsystem. Together, they produce output that
looks like this:

  Findings:
    F-001  CRITICAL sqli:sql_injection  (param=q)
           - [sql_single_quote] db_error: SQLITE error signature matched
           - [sql_single_quote] status_change: Baseline 200 -> Payload 500
           - [sql_or_1eq1]      length_anomaly: Response 3.2x baseline
           - [sql_union_null]   status_change: Baseline 200 -> Payload 500
           - [sql_union_null]   length_anomaly: Response 8.3x baseline
           - [sql_sleep_mysql]  status_change: Baseline 200 -> Payload 500
           - [sql_sleep_pg]     status_change: Baseline 200 -> Payload 500
           - [sql_sleep_pg]     length_anomaly: Response 4.7x baseline

    F-002  LOW      sqli:reflection  (param=q)
           - [sql_single_quote] payload_reflected: Payload reflected (raw)

Eleven signals. One bug. Every piece of evidence for F-001 points at
the same SQL injection, so JANISSARY groups them. F-002 is a different
root cause — reflection — and stays separate.

The first version of this scan reported nine findings. A reviewer looking
at the summary saw 9 findings and moved on. Now they see 2 groups and
read the one that matters.

What we gave up

We gave up speed.

A baseline-gated scan runs the target 3–5x more than a naive scanner. On a
100-endpoint target, that's the difference between a 4-minute scan and a
12-minute scan. We also gave up recall: when the baseline is unstable, we
disable the gates and miss findings rather than flood the report with
maybes.

We think that's the right trade. Here's the number that convinced us:

On two local vulnerable Flask endpoints (SQL injection and reflected
XSS), JANISSARY emitted 23 findings across 3 groups — and the
SQL-injection scan returned the identical finding set across two
back-to-back runs. A naive scan of the same endpoints produces 23
signals with no grouping, so the same bug shows up as 11 separate
findings, and triage starts at the wrong number.

That last word — reproducible — is the one most DAST tools can't claim.
Run the same scan twice and compare the output. If your tool's finding set
changes between identical runs, it isn't gating on differences, and you're
paying triage tax on every scan.

Does any of this hold at scale?

The Flask demo is a demo. Here's a harder test: OWASP Benchmark, the
standard open benchmark for DAST tools.

It contains 2,740 test cases across SQL injection, XSS, command
injection, path traversal, LDAP injection, XPath injection, weak
randomness, trust-boundary violations, and secure-cookie flags. Each
case is a servlet that either is or isn't vulnerable to a specific
class. The scanner has to figure out which without source access.

On the DAST-applicable subset — the corpus minus the crypto and hash
cases that only a source-aware tool can observe — JANISSARY scores:

F1        0.920
Precision 1.000
Recall    0.851
TP        984
FP        0
FN        172

Eighteen of the DAST-applicable categories score F1 1.000. Six more
score F1 0.95 or above. The weakest category is pathtraver, where the
F1 range across four subcategories is 0.049 to 0.308 — and there's a
separate write-up on why that's a property of the benchmark target,
not of the scanner: docs/pathtraver.md.

The number that matters most here is Precision 1.000. Not a single
false positive across the whole DAST-applicable corpus. That's the
design claim — a finding that isn't a difference is a guess — turned
into a measured outcome.

A scanner that emits zero false positives on a corpus this size isn't
"tuned." It's architected that way. Every oracle requires evidence of
a specific kind: a real database error signature, a session sink echo
of a per-scan random marker, a raw XSS payload past its trailing
character, a paired differential on a file-existence oracle. Nothing
fires on pattern match alone.

The 0.920 recall is the honest number. There are 172 cases JANISSARY
doesn't catch. 123 are the pathtraver target-cap; the rest are
scattered gaps across subcategories that we triage one at a time. The
full breakdown, including the cases we miss, is in BENCHMARK.md.
The point of running a benchmark is to learn where the tool fails,
not to have a number.

The run is reproducible:

python benchmark/track_a/run_with_recovery.py       --target-base http://127.0.0.1:8080/benchmark       --cases-csv benchmark/track_a/_chunks/full_p0.csv       --tools-env benchmark/track_a/tools_env_default.txt

Requires the OWASP Benchmark container; the script and the container
image are in the repo.

How do other tools do?

We ran Nuclei 3.11 and ZAP baseline against the same corpus (chunk 0,
550 URLs). Neither is a direct competitor to JANISSARY — they answer
different questions — but the comparison is instructive.

Tool	Benchmark TPs	Benchmark FPs	Findings on the corpus
JANISSARY	984 / 1156	0	2914
Nuclei 3.11	0	N/A	0
ZAP baseline	0	N/A	27

Nuclei's templates are keyed on known CVEs and version fingerprints.
The benchmark uses neither. Its 0 findings aren't a failure — they're a
measurement of scope. Nuclei is a vulnerability-discovery tool. The
OWASP Benchmark is not a discovery target.

ZAP baseline is passive-only: it spiders and reads. It found missing
security headers, missing anti-CSRF tokens, CSP weaknesses, and error
disclosures — all real signals about the benchmark's Tomcat setup,
none of them the injection classes the benchmark scores. Active
scanning (ZAP full) is the closer comparison, and its recall on
synthetic injection cases is historically below 20%. JANISSARY's
F1 0.920 is a specific outcome of the differential design, not a
category-wide win for "scanners."

The honest takeaway: these tools are complementary. Run Nuclei
for CVE coverage. Run ZAP for passive hygiene findings. Run JANISSARY

Update (Oct 18): I ran JANISSARY against a live target. It produced 39 false positives and missed a known SQLi. Here's the honest post-mortem: https://dev.to/janissary/0920-on-the-benchmark-zero-findings-on-juice-shop-3g4d-temp-slug-8188118/.
for reproducible differential detection on the injection classes.
They don't compete for the same triage slot.

What this means if you're evaluating DAST

Ask your current tool one question: if I run this scan twice, do I get
the same findings?

If the answer is no, differential gating isn't in it, and the tool is
producing guesses. That's not a moral judgment — a guess is useful if
you're hunting for leads. It's a problem if you're handing the output to a
client, an auditor, or your own backlog and expecting it to be actionable
without re-verification.

JANISSARY is the precision tool in your toolbox. It won't find everything.
It will only tell you what it can prove.

pip install janissary

github.com/Hetaireia/janissary

JANISSARY is a dual-use security testing tool. Run it only against
systems you own or have prior written permission to test. See the
Terms of Use.