SERP + News to Markdown for LLMs with the QuanticData API

#webscraping #llm #python #api

disclosure: I work on QuanticData.

Why this tutorial

Agents and local LLMs waste context on raw HTML. A better loop is: search → pick top links → read pages as clean Markdown. This post shows that loop with the QuanticData Data API (SERP + scrape) and the optional MCP server.

What you get

Google / Bing / DuckDuckGo search JSON (and news via search_type)
Page → Markdown from $0.0015 (render $0.0025)
SERP from $0.0009 ($0.90 per 1,000)
Failed calls are not billed; $2 free API usage per month
MCP: npx -y quanticdata-mcp or https://api.quanticdata.io/mcp

Register (no card): https://app.quanticdata.io/register
Docs: https://quanticdata.io/docs/ · https://quanticdata.io/llms.txt

Step 1 — SERP (web or news)

import os, requests

API = "https://api.quanticdata.io/v1"
H = {"Authorization": f"Bearer {os.environ['QUANTICDATA_API_KEY']}"}

serp = requests.post(f"{API}/scraper/serp", headers=H, json={
    "query": "web scraping API markdown LLM",
    "engine": "google",
    "search_type": "search",  # or "news"
    "country": "us",
    "lang": "en",
    "num": 10,
}, timeout=120).json()["payload"]

links = [row["link"] for row in serp.get("organic", [])[:5]]
print(links)

Step 2 — Scrape each URL to Markdown

pages = []
for url in links:
    r = requests.post(f"{API}/scraper/extract", headers=H, json={
        "url": url,
        "format": "markdown",
        "contentMode": "smart",
    }, timeout=120).json()
    if r.get("type") == "response":
        pages.append({
            "url": url,
            "title": r["payload"].get("title"),
            "markdown": r["payload"]["content"],
        })

Step 3 — MCP (optional)

QUANTICDATA_API_KEY=qd_live_… npx -y quanticdata-mcp
# hosted: https://api.quanticdata.io/mcp

Tools include scrape, search, search_and_read, map, crawl, batch.

Honest limits

Respect site ToS and robots where they apply; public pages / legitimate research.
List prices from QuanticData docs as of 2026-10; re-check https://quanticdata.io/pricing/ before budgeting.
No invented benchmarks — measure on your queries.

Wrap-up

Search JSON + Markdown pages is the smallest useful web toolset for agents. If you try the script, tell me what broke.

DEV Community