DEV Community

Devil Scrapes
Devil Scrapes

Posted on

NVD gives you 5 requests per 30 seconds, and CVSS in four schemas

Quick answer

NVD is free, authoritative, and rate-limited to 5 requests per 30 seconds if you have no API key — which is why most scripts against it die halfway through a backfill. The NVD CVE Scraper throttles to that limit, resolves CVSS across four schema versions, and flattens each vulnerability into one sortable row at $2.20 per 1,000 CVEs.

The three things that break a naive NVD client 🛡️

The rate limit is stricter than it looks. Unauthenticated clients get 5 requests per rolling 30 seconds. A loop that pages as fast as it can will succeed for a page or two and then collect 403s for the rest of the run. The failure arrives after you have started trusting the output, so a backfill quietly stops at a fraction of the corpus. This Actor spaces pages 6.5 seconds apart and backs off on 403 rather than treating it as fatal.

CVSS is not one field. NVD publishes scores under cvssMetricV40, cvssMetricV31, cvssMetricV30 and cvssMetricV2, and a single CVE often carries several at once. Read the wrong key and you get an outdated score; read the first key present and you get whichever the JSON happened to order first. This Actor takes the newest schema available per CVE and records which one in a cvss_version column, so a severity sort is comparing like with like.

There is a trap inside the trap: CVSS v2 puts baseSeverity outside cvssData, while v3 and later put it inside. A parser written against v3 returns null severity for every pre-2015 CVE. Both shapes are handled.

The date window is capped at 120 days. NVD rejects a wider publication range outright. The Actor validates the window before the run starts — and before you are charged — rather than letting the API refuse it mid-crawl.

The judgement call: unscored CVEs are kept 🔍

If you set a minimum CVSS score, it would be easy to drop every CVE without one. This Actor keeps them, deliberately.

A CVE with no published score is unassessed, not low severity. Newly disclosed vulnerabilities routinely sit in Awaiting Analysis for days before NVD attaches a score — which means a naive severity filter silently hides exactly the disclosures a security team most wants to see. The filter applies to CVEs that have a score; the rest come through and you can decide.

What you get per row

CVE ID and NVD URL, the English description, vulnStatus, publication and last-modified timestamps, CVSS version / score / severity / vector, attack vector, deduplicated CWE identifiers, the reporting source, and every reference URL published with the CVE.

Filter by keyword, by exact CPE 2.3 name when you know the product, by publication window, and by minimum score.

What it costs

Pay-per-event: a start fee plus $0.002 per CVE row. A thousand CVEs is about $2.20. No key to obtain, no subscription, and no charge for pages the API refuses.

Where it fits

Patch-cycle triage ranked by severity, a watchlist for CVEs naming a product in your stack, a vulnerability dashboard that does not need a commercial feed, or a historical exposure audit before adopting a dependency.

NVD CVE Scraper on Apify →

Top comments (0)