DEV Community

XSron Hou
XSron Hou

Posted on Originally published at scrapio.dev

Crawl a Competitor Site and Export to JSON

#webscraping #api #ai #showdev

Originally posted on the Scrapio blog — sharing here too.

Understanding a competitor's content strategy means knowing what pages they have, what topics they cover, and how they structure their information. Manual browsing doesn't scale. Scrapio's Crawl endpoint spiders an entire domain automatically and returns everything as structured data.

What you'll need

  • A Scrapio API key (any plan — Crawl isn't gated to a specific tier)
  • Python

Start a crawl

Pass seeds (an array of starting URLs), max_pages (up to 50), max_depth, and an optional extract object to pull structured data from each page.

import httpx
import json

API_KEY = "sk-..."
BASE = "https://api.scrapio.dev"

resp = httpx.post(
    f"{BASE}/v1/crawl",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "seeds": ["https://competitor.com"],
        "max_pages": 50,
        "max_depth": 4,
        "same_domain_only": True,
        "output": ["json"],
        "extract": {
            "mode": "schema",
            "schema": {
                "title": "string",
                "description": "string",
                "word_count": "integer",
                "publish_date": "string",
            },
        },
    },
    timeout=300,
)
resp.raise_for_status()
result = resp.json()
Enter fullscreen mode Exit fullscreen mode

The crawl runs synchronously and returns when complete.

Inspect the results 

pages = result["result"]["pages"]
summary = result["result"]["summary"]

print(f"Pages fetched:    {summary['pages_fetched']}")
print(f"Pages succeeded:  {summary['pages_succeeded']}")
print(f"Pages failed:     {summary['pages_failed']}")
Enter fullscreen mode Exit fullscreen mode

Each page in pages looks like:

{
  "url": "https://competitor.com/blog/web-scraping-guide",
  "depth": 1,
  "status": "completed",
  "outputs": {
    "json": {
      "title": "The Complete Guide to Web Scraping",
      "description": "Everything you need to know about extracting data from websites.",
      "word_count": 3842,
      "publish_date": "2026-03-15"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Export to JSON 

with open("competitor_content.json", "w") as f:
    json.dump(pages, f, indent=2)

print(f"Exported {len(pages)} pages")
Enter fullscreen mode Exit fullscreen mode

Analyze with pandas 

import pandas as pd

rows = [
    {
        "url": p["url"],
        "depth": p["depth"],
        "status": p["status"],
        **p.get("outputs", {}).get("json", {}),
    }
    for p in pages
    if p["status"] == "completed"
]
df = pd.DataFrame(rows)

print("Average word count:", df["word_count"].mean())
print("\nTop 10 longest pages:")
print(df.nlargest(10, "word_count")[["url", "word_count"]])
Enter fullscreen mode Exit fullscreen mode

What to look for

  • Content gaps — topics they cover that you don't
  • Word count distribution — how long are their top-performing pieces?
  • Update frequency — filter by publish_date to see how often they publish
  • URL structure — how is their site organized? What categories do they prioritize?

Next steps

Top comments (0)