DEV Community

CyberSpec
CyberSpec

Posted on

Why my vulnerability scanner says "likely fail" but never "pass"

I build a small external vulnerability scanner aimed at UK firms preparing for Cyber Essentials (CE), the government-backed baseline certification. The most important design decision in the product was one word I refused to ship: "pass".

The setup

Cyber Essentials has five technical controls. When I mapped them against what a scanner sitting outside your network can actually observe, I got this:

Control Visible from outside?
Firewalls / internet gateways Mostly: which services are reachable at all
Secure configuration Mostly: exposed admin panels, unnecessary services
Security update management Mostly: known-vulnerable and end-of-life versions
User access control No
Malware protection No

Three out of five, roughly. The temptation is to show a green tick on the three you can see. That would be wrong.

Why a scan can fail you but never pass you

An external scan can find a reason to fail. Under the NCSC's CE Plus test spec, a reachable service with a vulnerability scoring CVSS 7.0 or higher fails the external test outright. If my scanner sees one, that's a real, evidence-backed "this will likely fail".

But "no issues found" is not "pass". Plenty of things that matter are invisible from outside:

  • default passwords that were never changed
  • whether logins lock out after repeated failures
  • patching on internal machines
  • whether that exposed admin panel has MFA in front of it

So the readiness states in the product are:

  • likely fail: evidence of a fail condition
  • review: an exposure a human has to justify (e.g. a reachable admin interface needs a documented business need plus MFA or an IP allow list; the scan can see the panel, never the documentation)
  • no issues found: we looked and found nothing, which is narrower than "fine"
  • no scan yet: different from "no issues found", and worth keeping separate

The two controls a scan can't see at all are customer-attested: the user records a statement, versioned against the scheme version, and the UI never pretends it measured anything.

Simplified, the patching rule looks like this:

type Status = "likely_fail" | "review" | "no_issues_found" | "no_scan_yet";

function updateManagementStatus(findings: Finding[], hasScan: boolean): Status {
  if (!hasScan) return "no_scan_yet";
  const failing = findings.filter(
    (f) => f.open && (f.checkType === "eol" || (f.cve && (f.cvss ?? 0) >= 7.0)),
  );
  return failing.length ? "likely_fail" : "no_issues_found";
}
Enter fullscreen mode Exit fullscreen mode

Note what's missing: there is no branch that returns "pass".

A rule people mix up

"CVSS 7+ with a fix available for more than 14 days" is often quoted as the CE patching rule. In the CE Plus test spec, that specific check is sub-test 2.1, the authenticated scan of sampled devices, not the external scan. If you're building tooling against a compliance scheme, read the primary documents (NCSC publishes them as PDFs) and not vendor blog summaries. I got one claim wrong from general knowledge before checking them.

The takeaway for anyone building compliance tooling

Your product's vocabulary is a promise. A customer who sees "pass" will act on it, and then fail an audit. Listing what you couldn't check honestly is a feature, not an apology for a missing one.


Disclosure: I'm the founder of CyberSpec, the scanner described here. It's a readiness tool, not a certification body, and isn't affiliated with IASME or NCSC. Happy to answer questions in the comments.

Top comments (0)