0.920 on the benchmark. Zero findings on Juice Shop.

Three days ago I published JANISSARY's benchmark results: F1 0.920, precision 1.000 on the OWASP Benchmark DAST-applicable subset. Zero false positives across 2,258 cases.

Today I ran it against OWASP Juice Shop. It found 39 vulnerabilities. All 39 were wrong. And it missed the SQL injection I already knew was in the search endpoint.

Both of those things are true. This post is about why.

What the benchmark actually promises

The benchmark tests detection on a corpus of Java servlets. Each servlet either is or isn't vulnerable to a specific class. The scanner gets no source access. JANISSARY scored precision 1.000 — not a single finding on the corpus that wasn't a real vulnerability.

That number is real. It's reproducible. The run is in the repo.

But a benchmark measures one thing: how the tool behaves on the corpus. It doesn't promise the tool behaves the same way on a Node app.

That distinction is the entire post.

What happened on Juice Shop

Juice Shop is a Node/Express single-page app. It's the canonical intentionally-vulnerable target — thousands of people use it to practice web exploitation.

I ran:

janissary agent http://localhost:3000 --attack-confirm

Output:

Platforms:       wordpress
Modules run:     xmlrpc, admin
Modules skipped: scan
New findings:    39

Two things are wrong before you even look at the findings. Juice Shop isn't WordPress. And the scan module — the one with the differential oracles that produced the benchmark result — never ran.

The platform fingerprinter classified a Node app as WordPress. The agent then ran WordPress modules. The injection oracles sat this one out.

The 39 findings

All 39 had the same shape:

category:        admin
finding_type:    exposed-panel
discriminator:   /wp-admin/
detail:          wordpress /wp-admin/ -> HTTP 200
response_status: 200

Juice Shop's SPA router returns HTTP 200 and index.html for any path, including paths that don't exist. The admin module checked the status code, saw 200, and reported an exposed WordPress admin panel.

On a single-page app, every path returns 200. Every path was reported.

This is exactly the class of false positive JANISSARY's differential gating is designed to prevent. It fired anyway — because the admin module doesn't go through the differential gate. It's a status-code check wearing a scanner's clothing.

The miss

Juice Shop's /rest/products/search?q= endpoint is a documented SQL injection challenge. It's the first thing anyone tries.

janissary scan -u "http://localhost:3000/rest/products/search?q=apple" -p q

Output:

Total requests:  40
Findings:        0 in 0 groups

The scan ran. The baseline was stable. The WAF probe was clean. Nothing fired.

Here's the query Juice Shop builds internally:

SELECT * FROM Products
WHERE ((name LIKE '%apple%' OR description LIKE '%apple%')
AND deletedAt IS NULL)

To inject, you have to close two parens first:

apple')) UNION SELECT ...--

JANISSARY's payload set is benchmark-shaped. It sends ', ' OR '1'='1, and similar. Those are correct for the benchmark's servlets, which do simple string concatenation. On Juice Shop, ' produces a syntax error that the app swallows, returning an empty result set. No differential delta. No finding.

The payload set doesn't know about framework-wrapped queries.

What this says about benchmark numbers

The 0.920 is not fake. It measures what it says it measures: how JANISSARY performs on a corpus of Java servlets with simple injection points.

What it doesn't measure:

• Non-Java platforms
• Queries wrapped in parentheses or built by ORMs
• SPAs with catch-all routing
• Endpoints behind authentication
• Any error output that doesn't match the benchmark's signature set

Those aren't edge cases. They're most of the web.

So the honest reading of "F1 0.920" is: "this tool performs at 0.920 on a well-defined class of applications with well-defined vulnerability shapes." The class is narrower than "web applications." It's roughly "Java servlet-style apps with simple injection points."

That's still a real result. It's just not the result I implied by publishing the number without qualification.

What I'm doing about it

Three issues, all public:

• #1 — admin module reports false positives on SPA catch-all routing
• #3 — SQLi payload set lacks framework-specific variants (paren-closing, comment terminators)

• #2 — scan aborts on HTTP 401 without reporting auth-required state
  None of these are launch-killers. All of them are fixable. The fixes are specific, small, and testable:

• The admin probe should verify response body shape, not just status

• The payload library should include ')), --, /* */, and quote-stacking variants

• 401 responses should report "endpoint requires authentication" instead of aborting silently

Each fix has a reproduction case from Juice Shop. Each can be verified against the same target.

The honest version of the claim

If you're evaluating JANISSARY on the strength of the benchmark number, here's what you should actually believe:

• On applications that look like the OWASP Benchmark's servlets, it performs at F1 0.920 with zero false positives. That's measured.
• On applications that don't, its behavior is uncharacterized. As of today, the first live-target run produced false positives from one module and misses from another.

The tool's whole thesis is that findings should be reproducible differences, not pattern matches. That thesis has to apply to the tool's own claims. The benchmark number was a pattern match on a corpus. The Juice Shop run is the difference.

---

JANISSARY is open source (Apache 2.0). Repo: https://github.com/Hetaireia/janissary. Issues: https://github.com/Hetaireia/janissary/issues. Benchmark methodology: BENCHMARK.md.