How to Scrape and Parse CVE Data from NVD API v2

The NVD (National Vulnerability Database) changed its public API in 2023, and many existing scrapers broke without warning. If you're building a vulnerability tracker, a security dashboard, or an automated patch prioritization tool, you need to work with API v2 — the old v1 endpoints are permanently shut down. This guide covers authentication, pagination, safe JSON parsing, and the rate limit traps most tutorials skip.

What Changed in NVD API v2

The old API (v1) required no authentication and returned a relatively flat JSON structure. API v2 is different in three important ways:

API key required for usable rate limits. Without a key, you're capped at 5 requests per 30 seconds. With a free NVD API key you get 50 requests per 30 seconds — a 10x difference that matters when you're backfilling months of CVE history. Register at nvd.nist.gov/developers/request-an-api-key.

Pagination is mandatory. The API returns at most 2000 results per call. For any real query you must use startIndex to walk through pages.

The JSON schema changed. CVE data now lives under vulnerabilities[].cve rather than result.CVE_Items. Fields like CVSS scores are split across three separate metric versions (v2, v3.0, v3.1) and any of them may be absent for recently published CVEs.

Fetching CVEs: Building the Request

The base URL is https://services.nvd.nist.gov/rest/json/cves/2.0. The API key goes in a header (apiKey), not a query parameter. Key filter parameters:

• keywordSearch — full-text search across CVE descriptions
• pubStartDate / pubEndDate — publication date range; must use ISO 8601 with explicit timezone offset: 2024-01-01T00:00:00.000+00:00
• cvssV3Severity — filter by LOW, MEDIUM, HIGH, or CRITICAL
• lastModStartDate / lastModEndDate — for incremental syncs, fetch only CVEs modified since your last pull

import time
import requests
from datetime import datetime

NVD_BASE = "https://services.nvd.nist.gov/rest/json/cves/2.0"
API_KEY = "your-nvd-api-key"  # set to None to test without a key

def fetch_cves(
    keyword: str | None = None,
    severity: str | None = None,
    start_date: datetime | None = None,
    end_date: datetime | None = None,
    start_index: int = 0,
    results_per_page: int = 500,
) -> dict:
    headers = {"apiKey": API_KEY} if API_KEY else {}
    params: dict = {
        "startIndex": start_index,
        "resultsPerPage": min(results_per_page, 2000),
    }
    if keyword:
        params["keywordSearch"] = keyword
    if severity:
        params["cvssV3Severity"] = severity.upper()
    if start_date:
        params["pubStartDate"] = start_date.strftime("%Y-%m-%dT%H:%M:%S.000+00:00")
    if end_date:
        params["pubEndDate"] = end_date.strftime("%Y-%m-%dT%H:%M:%S.000+00:00")

    resp = requests.get(NVD_BASE, headers=headers, params=params, timeout=30)
    resp.raise_for_status()
    return resp.json()

Parsing the Response Without Breaking

Naively accessing data["vulnerabilities"][0]["cve"]["metrics"]["cvssMetricV31"][0]["cvssData"]["baseScore"] will blow up on roughly 30% of CVEs. CVSS data is optional, versioned, and sometimes an empty list even when the key exists.

from dataclasses import dataclass

@dataclass
class CVERecord:
    cve_id: str
    description: str
    published: str
    severity: str
    cvss_score: float | None
    references: list[str]

def parse_cve(raw: dict) -> CVERecord:
    cve = raw["cve"]

    # prefer English, fall back to first available
    descriptions = cve.get("descriptions", [])
    description = next(
        (d["value"] for d in descriptions if d.get("lang") == "en"),
        descriptions[0]["value"] if descriptions else "",
    )

    # try CVSS v3.1 -> v3.0 -> v2.0 in order of preference
    metrics = cve.get("metrics", {})
    cvss_score: float | None = None
    severity = "UNKNOWN"
    for metric_key in ("cvssMetricV31", "cvssMetricV30", "cvssMetricV2"):
        entries = metrics.get(metric_key, [])
        if entries:
            data = entries[0].get("cvssData", {})
            cvss_score = data.get("baseScore")
            severity = data.get("baseSeverity", "UNKNOWN")
            break

    references = [r["url"] for r in cve.get("references", []) if "url" in r]

    return CVERecord(
        cve_id=cve["id"],
        description=description,
        published=cve.get("published", ""),
        severity=severity,
        cvss_score=cvss_score,
        references=references,
    )

The key pattern: iterate over metric versions in descending preference order and break on the first non-empty list. This gives you the most precise score available without ever crashing on a freshly published CVE that has not been scored yet.

Pagination and Rate Limit Handling

A search for "OpenSSL" or "Apache" returns thousands of CVEs. You need to paginate and sleep between requests to stay within the rate limit ceiling.

def fetch_all_cves(
    keyword: str,
    severity: str | None = None,
    max_results: int | None = None,
) -> list[CVERecord]:
    sleep_s = 0.7 if API_KEY else 6.1  # respect rate limits
    page_size = 500
    start = 0
    results: list[CVERecord] = []

    while True:
        data = fetch_cves(
            keyword=keyword,
            severity=severity,
            start_index=start,
            results_per_page=page_size,
        )
        total = data["totalResults"]
        batch = [parse_cve(v) for v in data.get("vulnerabilities", [])]
        results.extend(batch)
        print(f"  {len(results)} / {total}")

        if max_results and len(results) >= max_results:
            break
        if start + page_size >= total:
            break

        start += page_size
        time.sleep(sleep_s)

    return results

For production pipelines, do not re-fetch everything from scratch on every run. Use lastModStartDate set to your last successful sync timestamp. The NVD updates existing CVE records — re-scoring, adding references, changing status — so you need this to catch edits to records you already ingested, not just new ones.

Caching in SQLite for Team Use

If more than one person needs CVE data, add a local SQLite cache so queries do not hit the NVD on every call. Index on severity and published to keep filtering fast.

import sqlite3, json

def init_db(path: str = "cves.db") -> sqlite3.Connection:
    conn = sqlite3.connect(path)
    conn.execute(
        "CREATE TABLE IF NOT EXISTS cves ("
        "cve_id TEXT PRIMARY KEY, description TEXT, published TEXT,"
        "severity TEXT, cvss_score REAL, references_json TEXT,"
        "fetched_at TEXT DEFAULT (datetime('now')))"
    )
    conn.execute("CREATE INDEX IF NOT EXISTS idx_severity ON cves(severity)")
    conn.execute("CREATE INDEX IF NOT EXISTS idx_published ON cves(published)")
    conn.commit()
    return conn

def upsert_cves(conn: sqlite3.Connection, records: list) -> int:
    rows = [
        (r.cve_id, r.description, r.published, r.severity,
         r.cvss_score, json.dumps(r.references))
        for r in records
    ]
    conn.executemany(
        "INSERT OR REPLACE INTO cves"
        " (cve_id, description, published, severity, cvss_score, references_json)"
        " VALUES (?, ?, ?, ?, ?, ?)",
        rows,
    )
    conn.commit()
    return len(rows)

With this in place, run a nightly job that fetches the previous day's CVEs using lastModStartDate and upserts them. Your team can query the local database — filtering by severity, CVSS score range, or description keyword — without every lookup costing a network roundtrip to the NVD.

One caveat before wiring CVE data to automated alerting: CVSS base scores do not encode your environment. A CRITICAL vulnerability in a library you do not ship matters less than a MEDIUM in your authentication layer. A structured triage process that maps CVEs to your actual attack surface is what turns raw data into actionable work. We cover that workflow step by step in our security hardening checklists.

The Takeaway

NVD API v2 gives you reliable, structured access to the full CVE catalog, but it requires more defensive coding than the old API. Three things that will save you time:

1. Get an API key first — 10x rate limit for free, takes two minutes.
2. Never trust CVSS fields to exist — fall back gracefully across metric versions.
3. Use lastModStartDate for incremental syncs — otherwise you re-fetch tens of thousands of records to find a few hundred changes.

From here you can wire CVE records to your SBOM, fire Slack alerts when a new CRITICAL lands on software in your stack, or build a scoring layer on top that factors in your actual exposure context.

---

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.