DEV Community: XSron Hou

Crawl a Competitor Site and Export to JSON

XSron Hou — Mon, 07 Sep 2026 03:00:00 +0000

Originally posted on the Scrapio blog — sharing here too.

Understanding a competitor's content strategy means knowing what pages they have, what topics they cover, and how they structure their information. Manual browsing doesn't scale. Scrapio's Crawl endpoint spiders an entire domain automatically and returns everything as structured data.

What you'll need

A Scrapio API key (any plan — Crawl isn't gated to a specific tier)
Python

Start a crawl

Pass seeds (an array of starting URLs), max_pages (up to 50), max_depth, and an optional extract object to pull structured data from each page.

import httpx
import json

API_KEY = "sk-..."
BASE = "https://api.scrapio.dev"

resp = httpx.post(
    f"{BASE}/v1/crawl",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "seeds": ["https://competitor.com"],
        "max_pages": 50,
        "max_depth": 4,
        "same_domain_only": True,
        "output": ["json"],
        "extract": {
            "mode": "schema",
            "schema": {
                "title": "string",
                "description": "string",
                "word_count": "integer",
                "publish_date": "string",
            },
        },
    },
    timeout=300,
)
resp.raise_for_status()
result = resp.json()

The crawl runs synchronously and returns when complete.

Inspect the results

pages = result["result"]["pages"]
summary = result["result"]["summary"]

print(f"Pages fetched:    {summary['pages_fetched']}")
print(f"Pages succeeded:  {summary['pages_succeeded']}")
print(f"Pages failed:     {summary['pages_failed']}")

Each page in pages looks like:

{
  "url": "https://competitor.com/blog/web-scraping-guide",
  "depth": 1,
  "status": "completed",
  "outputs": {
    "json": {
      "title": "The Complete Guide to Web Scraping",
      "description": "Everything you need to know about extracting data from websites.",
      "word_count": 3842,
      "publish_date": "2026-03-15"
    }
  }
}

Export to JSON

with open("competitor_content.json", "w") as f:
    json.dump(pages, f, indent=2)

print(f"Exported {len(pages)} pages")

Analyze with pandas

import pandas as pd

rows = [
    {
        "url": p["url"],
        "depth": p["depth"],
        "status": p["status"],
        **p.get("outputs", {}).get("json", {}),
    }
    for p in pages
    if p["status"] == "completed"
]
df = pd.DataFrame(rows)

print("Average word count:", df["word_count"].mean())
print("\nTop 10 longest pages:")
print(df.nlargest(10, "word_count")[["url", "word_count"]])

What to look for

Content gaps — topics they cover that you don't
Word count distribution — how long are their top-performing pieces?
Update frequency — filter by publish_date to see how often they publish
URL structure — how is their site organized? What categories do they prioritize?

Next steps

Try the Full Site Crawl to JSON template for the complete recipe
See Site Map Discovery to map a domain's URL structure without downloading full content
Read the Crawl API docs

Schedule a Weekly Price Monitor in 10 Minutes

XSron Hou — Mon, 31 Aug 2026 03:00:00 +0000

Originally posted on the Scrapio blog — sharing here too.

Price monitoring is a common automation need — whether you're tracking a competitor, waiting for a deal, or monitoring supplier costs. The traditional path involves a server, a cron job, a scraper, and a notification system. With Scrapio you only need the last two.

Scrapio now also has native Monitors — set the URL, the fields to extract, and which ones to watch from the dashboard, and Scrapio owns the schedule, the "last value" comparison, and the webhook delivery for you. The walkthrough below is still useful if you want the cron trigger and comparison logic running in your own infrastructure (e.g. to chain into other systems), but for most teams the native Monitor is the faster path — skip to Next steps to try it.

What you'll need

A Scrapio API key
A cron trigger (GitHub Actions, cron-job.org, or any scheduler)
A webhook URL to receive notifications (use webhook.site for testing)

The approach

The pattern is simple:

A cron trigger fires on a schedule (e.g. every Monday at 9am)
It calls Scrapio to extract the current price from the product page
It compares against the last recorded price
If the price dropped, it sends a notification

Step 1 — Fetch the price with Scrapio

Use the extract parameter to pull just the fields you care about:

import httpx

API_KEY = "sk-..."

def get_price(url: str) -> dict:
    resp = httpx.post(
        "https://api.scrapio.dev/v1/fetch",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={
            "url": url,
            "output": ["json"],
            "extract": {
                "mode": "schema",
                "schema": {
                    "price": "number",
                    "availability": "string",
                },
            },
            "render_js": True,
        },
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json()["outputs"]["json"]

Step 2 — Compare and notify

import json
import os
import httpx

PRICE_FILE = "last_price.json"
WEBHOOK_URL = "https://your-webhook-url.com/price-alert"
PRODUCT_URL = "https://example.com/products/iphone-15"

def load_last_price() -> float | None:
    if os.path.exists(PRICE_FILE):
        return json.loads(open(PRICE_FILE).read()).get("price")
    return None

def save_price(price: float) -> None:
    open(PRICE_FILE, "w").write(json.dumps({"price": price}))

def run():
    current = get_price(PRODUCT_URL)
    price = current["price"]
    last = load_last_price()

    if last is not None and price < last:
        httpx.post(WEBHOOK_URL, json={
            "message": f"Price dropped from ${last:.2f} to ${price:.2f}",
            "url": PRODUCT_URL,
            "price": price,
        })

    save_price(price)

run()

Step 3 — Schedule with GitHub Actions

Create .github/workflows/price-monitor.yml:

name: Price Monitor
on:
  schedule:
    - cron: "0 9 * * 1"  # Every Monday at 9am UTC
  workflow_dispatch:

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install httpx
      - run: python monitor.py
        env:
          SCRAPIO_API_KEY: ${{ secrets.SCRAPIO_API_KEY }}

GitHub Actions is free for public repos and the free tier covers hundreds of runs per month for private ones. No server required.

Cron expressions quick reference

Expression	Meaning
`0 9 * * 1`	Every Monday at 9am UTC
`0 /6 * *`	Every 6 hours
`0 8 * * *`	Every day at 8am UTC
`/30 * * *`	Every 30 minutes

Next steps

Try the native Monitors dashboard — no cron trigger or webhook server to host
See the Native Price Monitoring template for a step-by-step walkthrough
Monitoring content beyond price? See Website Change Detector — or use a native monitor with "watch": {"mode": "content"} for the same thing without hosting anything
Prefer to run it yourself? Try the Price Drop Alert template for the GitHub Actions recipe
Read the Fetch API docs

Take Screenshots of Any Website via API

XSron Hou — Mon, 31 Aug 2026 03:00:00 +0000

Originally posted on the Scrapio blog — sharing here too.

Screenshots via API have more use cases than you might expect: visual regression testing, monitoring landing pages for unauthorized changes, generating OG image previews, archiving pages, and building internal dashboards with live page previews.

What you'll need

A Scrapio API key
curl or Python

Take a basic screenshot

curl -X POST https://api.scrapio.dev/v1/fetch \
  -H "Authorization: Bearer sk-..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "output": ["screenshot"],
    "render_js": true
  }'

Screenshot output always requires render_js: true — without it, the request is routed to a non-browser executor that can't capture a screenshot and the call fails.

The response contains a screenshot object with a url field pointing to the captured image:

{
  "request_id": "req_abc123",
  "mode": "inline",
  "status": "completed",
  "outputs": {
    "screenshot": {
      "url": "https://storage.scrapio.dev/screenshots/req_abc123.png"
    }
  },
  "usage": { "credits": 2 }
}

Capture a mobile viewport

Pass device: "mobile" to render the page at a mobile viewport:

import httpx

resp = httpx.post(
    "https://api.scrapio.dev/v1/fetch",
    headers={"Authorization": "Bearer sk-..."},
    json={
        "url": "https://example.com",
        "output": ["screenshot"],
        "render_js": True,
        "device": "mobile",
    },
    timeout=30,
)
screenshot_url = resp.json()["outputs"]["screenshot"]["url"]
print(screenshot_url)

device accepts "desktop" (default), "mobile", or "tablet".

Wait for JavaScript content to finish loading

render_js: true is required for every screenshot request. For pages that load content asynchronously, pair it with wait_for:

resp = httpx.post(
    "https://api.scrapio.dev/v1/fetch",
    headers={"Authorization": "Bearer sk-..."},
    json={
        "url": "https://example.com/dashboard",
        "output": ["screenshot"],
        "render_js": True,
        "wait_for": {"network_idle": True},
    },
    timeout=30,
)
screenshot_url = resp.json()["outputs"]["screenshot"]["url"]

wait_for: {network_idle: true} tells Scrapio to hold until all network requests settle before capturing — useful for charts and dashboards that load data asynchronously.

Combine screenshot with markdown

You can request multiple outputs in a single call:

resp = httpx.post(
    "https://api.scrapio.dev/v1/fetch",
    headers={"Authorization": "Bearer sk-..."},
    json={
        "url": "https://example.com/article",
        "output": ["screenshot", "markdown"],
        "render_js": True,
    },
    timeout=30,
)
outputs = resp.json()["outputs"]
screenshot_url = outputs["screenshot"]["url"]
text = outputs["markdown"]

Visual monitoring workflow

Download the screenshot and diff against a baseline using Pillow:

import httpx
from PIL import Image, ImageChops
import io

def screenshot_url(url: str) -> Image.Image:
    resp = httpx.post(
        "https://api.scrapio.dev/v1/fetch",
        headers={"Authorization": "Bearer sk-..."},
        json={"url": url, "output": ["screenshot"], "render_js": True},
        timeout=30,
    )
    img_url = resp.json()["outputs"]["screenshot"]["url"]
    img_bytes = httpx.get(img_url).content
    return Image.open(io.BytesIO(img_bytes))

def has_visual_change(baseline: Image.Image, current: Image.Image) -> bool:
    diff = ImageChops.difference(baseline, current)
    return diff.getbbox() is not None

Next steps

Read the Fetch API docs for all options
See Website Change Detector for content-based monitoring

How to Scrape Google Search Results with an API

XSron Hou — Mon, 24 Aug 2026 03:01:00 +0000

Originally posted on the Scrapio blog — sharing here too.

Google Search is one of the most valuable data sources on the internet, and one of the hardest to scrape reliably. Google detects and blocks bots aggressively — rate limits, CAPTCHAs, and IP bans make direct scraping fragile.

Scrapio routes search requests through a specialized proxy layer that mirrors real browser behavior. You get clean, structured results without any of the blocking.

What you'll need

A Scrapio API key on a Pro plan or higher (Google Search isn't available on Free or Starter)
curl or Python

Basic search request

The Google Search endpoint is a GET request — pass parameters as query strings:

curl "https://api.scrapio.dev/v1/google/search?search=best+web+scraping+api+2026" \
  -H "Authorization: Bearer sk-..."

Response:

{
  "organic_results": [
    {
      "position": 1,
      "title": "Top 10 Web Scraping APIs Compared (2026)",
      "url": "https://example.com/web-scraping-apis",
      "description": "We tested 10 web scraping APIs on speed, reliability, and price...",
      "domain": "example.com"
    }
  ],
  "questions": [
    "Is web scraping legal?",
    "What is the difference between scraping and crawling?",
    "How do I scrape Google without getting blocked?"
  ],
  "related_searches": [...]
}

Paginate results

import httpx

API_KEY = "sk-..."

def get_serp_page(query: str, page: int = 1) -> dict:
    return httpx.get(
        "https://api.scrapio.dev/v1/google/search",
        headers={"Authorization": f"Bearer {API_KEY}"},
        params={
            "search": query,
            "page": page,
        },
        timeout=30,
    ).json()

# Collect first 3 pages of results
all_organic = []
for page in range(1, 4):
    results = get_serp_page("web scraping api", page)
    all_organic.extend(results.get("organic_results", []))

print(f"Collected {len(all_organic)} results")

Localize results

Pass country_code (ISO 3166-1 alpha-2) and language to get localized SERPs:

httpx.get(
    "https://api.scrapio.dev/v1/google/search",
    headers={"Authorization": f"Bearer {API_KEY}"},
    params={
        "search": "web scraping api",
        "country_code": "de",
        "language": "de",
    },
    timeout=30,
).json()

Search type

Use search_type to query different Google verticals:

Value	Description
`classic`	Standard web search (default)
`news`	Google News results
`images`	Google Images
`lens`	Google Lens (visual search)
`ai_mode`	Google AI Mode results
`ads`	Google Ads results

shopping and maps aren't supported — Google blocks them on every provider path this API has access to, so they're rejected at validation rather than silently failing.

params={"search": "wireless headphones", "search_type": "images"}

Use cases

SEO rank tracking — monitor your keyword positions daily
Competitor research — see what's ranking for your target keywords
Market research — extract PAA boxes to understand user intent
Content strategy — find gaps in top-ranking pages

Next steps

Try the Google SERP to JSON template for a ready-to-run recipe
See Google News Monitor for tracking news coverage by keyword
Read the Google Search API docs

How to Extract Structured JSON from Any Website

XSron Hou — Mon, 17 Aug 2026 03:00:00 +0000

Originally posted on the Scrapio blog — sharing here too.

Every data pipeline eventually hits the same wall: the information you need is on a web page, but it comes back as a blob of HTML. Writing CSS selectors works until the site redesigns. Regex works until it doesn't.

Scrapio's extract parameter takes a different approach — you describe the shape of the data you want, and the API figures out how to pull it from any page.

What you'll need

A Scrapio API key
The URL of a page with structured content (product page, article, listing)
curl or Python

Define a schema

Pass output: ["json"] and an extract object with mode: "schema". The schema maps field names to their types — the API handles figuring out where those values live on the page.

curl -X POST https://api.scrapio.dev/v1/fetch \
  -H "Authorization: Bearer sk-..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products/wireless-headphones",
    "output": ["json"],
    "extract": {
      "mode": "schema",
      "schema": {
        "name": "string",
        "price": "number",
        "currency": "string",
        "availability": "string",
        "rating": "number",
        "review_count": "integer"
      }
    }
  }'

Response:

{
  "request_id": "req_abc123",
  "mode": "inline",
  "status": "completed",
  "outputs": {
    "json": {
      "name": "Sony WH-1000XM5 Wireless Headphones",
      "price": 279.99,
      "currency": "USD",
      "availability": "In Stock",
      "rating": 4.7,
      "review_count": 12483
    }
  },
  "usage": { "credits": 5 }
}

Schema and instruction extraction run through the LLM extractor by default (5 credits) whenever an LLM provider is configured on your account; it only falls back to the cheaper heuristic extractor (2 credits) if no provider is configured or the LLM call fails. Set "engine": "heuristic" in the extract object if you want to guarantee the 2-credit rate.

Extract with a natural-language instruction

For more complex extractions, use mode: "instruction" to describe what you want in plain English:

import httpx

resp = httpx.post(
    "https://api.scrapio.dev/v1/fetch",
    headers={"Authorization": "Bearer sk-..."},
    json={
        "url": "https://example.com/search?q=headphones",
        "output": ["json"],
        "extract": {
            "mode": "instruction",
            "instruction": "Extract all product listings. For each product return: title, price, URL, and rating.",
        },
    },
    timeout=30,
)
items = resp.json()["outputs"]["json"]
print(f"Found {len(items)} products")

Extract using CSS selectors

When you know the page structure, mode: "selectors" is the most precise option:

resp = httpx.post(
    "https://api.scrapio.dev/v1/fetch",
    headers={"Authorization": "Bearer sk-..."},
    json={
        "url": "https://example.com/products/headphones",
        "output": ["json"],
        "extract": {
            "mode": "selectors",
            "fields": {
                "title":  {"selector": "h1.product-title", "type": "text"},
                "price":  {"selector": ".price-now",       "type": "text"},
                "rating": {"selector": "[data-rating]",    "type": "attr", "attribute": "data-rating"},
            },
        },
    },
    timeout=30,
)
data = resp.json()["outputs"]["json"]

Tips

Keep schemas focused. Only request fields you need — the more fields, the harder the extraction.
Use render_js: true for pages that load prices or content via JavaScript.
Validate downstream. For production pipelines, validate the output shape before inserting into a database.

Next steps

Try the Product Price Extractor template
Read the Fetch API docs
See Full Site Crawl to JSON for bulk extraction across an entire domain

How to Feed Live Website Data into an LLM Agent

XSron Hou — Mon, 10 Aug 2026 03:00:00 +0000

Originally posted on the Scrapio blog — sharing here too.

LLM agents are only as good as the context you give them. Static knowledge cuts off at a training date — but most interesting tasks require knowing what a page says right now: a competitor's pricing, a news article, a product listing.

The bottleneck is usually getting clean text into the prompt. Raw HTML is noisy. Playwright is complex. Here's the simple path.

What you'll need

A Scrapio API key (free tier covers this example)
An LLM with a large enough context window (GPT-4o, Claude 3.5, Gemini 1.5 work well)
Python 3.10+

Fetch a page as markdown

Scrapio returns clean markdown by default. One API call strips navigation, ads, and boilerplate — leaving just the readable content.

import httpx

API_KEY = "sk-..."

def get_page_markdown(url: str) -> str:
    resp = httpx.post(
        "https://api.scrapio.dev/v1/fetch",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"url": url, "output": ["markdown"]},
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json()["outputs"]["markdown"]

Pass it to your agent

from openai import OpenAI

client = OpenAI()

def answer_about_page(url: str, question: str) -> str:
    page_content = get_page_markdown(url)

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": "You are a research assistant. Answer based only on the provided web page content.",
            },
            {
                "role": "user",
                "content": f"Page content:\n\n{page_content}\n\nQuestion: {question}",
            },
        ],
    )
    return response.choices[0].message.content

print(answer_about_page(
    "https://example.com/pricing",
    "What is the cheapest paid plan and what does it include?"
))

Why markdown beats raw HTML

Format	Tokens per page (avg)	Noise
Raw HTML	8,000–40,000	High (tags, scripts, styles)
Markdown	800–4,000	Low

Markdown is 5–10× smaller, which means lower cost and less context pressure on your model.

Handling JavaScript-heavy pages

If the target page loads content via JavaScript (SPAs, dashboards), set render_js to true:

json={"url": url, "output": ["markdown"], "render_js": True}

Scrapio spins up a headless browser, waits for the page to settle, and returns the rendered content.

Next steps

Try the Webpage to Markdown template for a ready-to-run example
Read the Fetch API docs for all available options
See RAG pipelines with crawled sites for bulk ingestion

Why Markdown Is Better Than HTML for LLM Context Windows

XSron Hou — Wed, 05 Aug 2026 15:43:09 +0000

Originally posted on the Scrapio blog — sharing here too.

When you're building a RAG pipeline or an LLM agent that reads web pages, the format you put into the context window matters more than most people realize. Raw HTML and clean markdown can represent the same information, but one costs 10× more in tokens and adds substantial noise that degrades model performance.

The problem with HTML

A typical product page might have 3,000 words of actual content. As raw HTML, that page is often 50,000–80,000 characters — mostly tags, class names, JavaScript, tracking pixels, and navigation menus.

LLMs tokenize all of it. You pay for every <div class="product-carousel__item--highlighted"> even though it carries no information your model needs.

Measured comparison

We ran 100 pages through both approaches and measured:

Metric	Raw HTML	Clean Markdown
Avg tokens per page	18,400	1,850
Tokens that are content	~12%	~94%
GPT-4o cost per 1,000 pages	~$91	~$9
Extraction accuracy (structured tasks)	71%	89%

The accuracy difference is real: LLMs perform better when they're not trying to reason through a wall of HTML noise.

What clean markdown looks like

Raw HTML snippet:

<div class="pdp-price-container" data-testid="price-wrapper">
  <span class="price--sale" aria-label="Sale price">
    <span class="price__currency">$</span>
    <span class="price__amount">279</span>
    <span class="price__cents">.99</span>
  </span>
</div>

Same content as markdown:

**Price:** $279.99

The markdown version is 22 tokens. The HTML version is 89 tokens — and that's a small excerpt. An entire page compounds this by 50–100×.

Getting clean markdown from any URL

curl -X POST https://api.scrapio.dev/v1/fetch \
  -H "Authorization: Bearer sk-..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/article", "output": ["markdown"]}'

Response:

{
  "request_id": "req_abc123",
  "mode": "inline",
  "status": "completed",
  "outputs": {
    "markdown": "# Article Title\n\nContent here..."
  },
  "usage": { "credits": 1 }
}

The markdown output strips navigation, ads, scripts, and layout elements. What remains is the readable content — headings, paragraphs, lists, tables, and code blocks.

JavaScript-rendered pages

SPAs and dynamic pages need a render step:

{
  "url": "https://example.com/dashboard",
  "output": ["markdown"],
  "render_js": true
}

Without render_js, you'd get the pre-hydration HTML — often an empty shell.

For RAG pipelines

When crawling large sites for a vector database, markdown is especially important because:

Chunking is cleaner — markdown structure (headings, paragraphs) provides natural split points
Embeddings are better — semantic models trained on clean text produce better vectors when the input isn't polluted with HTML syntax
Storage is cheaper — 10× smaller documents mean lower storage and fewer API calls to retrieve them

Next steps

Try the Webpage to Markdown template for a copy-paste implementation
See Batch Article Extractor for crawling and indexing entire sites
Read How to Feed Live Website Data into an LLM Agent