DEV Community: XSron Hou

Nine New Dedicated APIs: Airbnb, Reddit, TikTok, Target, Apple App Store, and More

XSron Hou — Mon, 28 Sep 2026 03:00:00 +0000

Originally posted on the Scrapio blog — sharing here too.

Scrapio v1.1.0 ships nine new dedicated site APIs in one release — the biggest expansion of target-shaped endpoints since launch. Every one of them is available today, gated at Starter plan or higher, with full support across all six SDKs and the MCP server.

What's new

Airbnb — search listings by location and dates, and pull structured detail for a single listing.
ChatGPT, Perplexity, and Gemini — send a prompt to any of the three major AI answer engines and get back structured text, citations, and (for ChatGPT and Perplexity) the sources behind the answer. Useful for AI-visibility and brand-mention tracking as these engines become a real discovery channel.
Bing Search — structured organic and paid SERP results, priced at the same tier as Google Search but available from Starter plan and up.
Reddit — our first social/UGC vertical: a post with its comments, a subreddit's post feed, or a user's activity, all as clean JSON.
TikTok — structured metadata and engagement stats for a video post: description, author, likes/comments/shares/plays, video and music info.
Target — a third general-retail API alongside Amazon and Walmart, with pickup/delivery/shipping localization.
Apple App Store — structured app listing data: name, price, category, rating, and developer.
YouTube Channel — a new endpoint on the existing YouTube API for channel metadata, subscriber/view counts, and recent uploads.

Why dedicated APIs

Every one of these sites is hard to scrape reliably on your own — JavaScript rendering, anti-bot defenses, and page structures that change without notice. A dedicated endpoint means you get a stable, target-shaped JSON contract instead of building and maintaining your own extraction logic against a moving target.

Available everywhere you already use Scrapio

SDKs — TypeScript, Python, Go, Ruby, PHP, and Java all have typed methods for every endpoint above.
MCP — 14 new tools are registered on the Scrapio MCP server, so any MCP-connected agent can use these directly.
Templates — example workflows for each new API are live on the templates page, including Airbnb listing search, tracking brand mentions in ChatGPT, and Target price monitoring.

Get started

Pick any endpoint above and try it with your existing API key — no new signup or add-on required, just a Starter plan or higher. See the pricing page for the full credit breakdown, or the full release notes for everything in v1.1.0.

Beyond Change Detection: Generating Visual Diff Images for Screenshot Alerts

XSron Hou — Mon, 28 Sep 2026 03:00:00 +0000

Originally posted on the Scrapio blog — sharing here too.

Boolean change detection ("did the page look different this run?") answers half the question. The other half — the one that actually matters when you're staring at a Slack alert at 2am — is what changed and where. A pixel-diff ratio over a threshold tells you to look; it doesn't tell you where to look. This guide covers generating an actual annotated diff image: the changed region boxed in red, ready to attach to a webhook payload or Slack message.

This builds on Screenshot Monitoring, which covers the scheduled capture-and-threshold-comparison side. This post is specifically about turning that comparison into a visual artifact a human can glance at instead of opening both screenshots side by side themselves.

What you'll need

A Scrapio API key
Python 3.10+ with httpx and Pillow installed (pip install httpx pillow)
Two screenshots already captured on a schedule (see Screenshot Monitoring for the capture + basic threshold-comparison side)

From boolean to bounding box

ImageChops.difference() (used in the boolean check) already computes a full diff image — the trick is using its bounding box to know where the change is, then drawing that box onto a copy of the current screenshot instead of discarding the diff once you have a yes/no answer:

from io import BytesIO
from PIL import Image, ImageChops, ImageDraw

def generate_diff_image(previous_bytes: bytes, current_bytes: bytes) -> bytes | None:
    prev = Image.open(BytesIO(previous_bytes)).convert("RGB")
    curr = Image.open(BytesIO(current_bytes)).convert("RGB")
    if prev.size != curr.size:
        curr = curr.resize(prev.size)

    diff = ImageChops.difference(prev, curr)
    bbox = diff.getbbox()
    if bbox is None:
        return None  # no change

    annotated = curr.copy()
    draw = ImageDraw.Draw(annotated)
    # Pad the box slightly so the highlight isn't drawn flush against the
    # changed pixels themselves -- easier to actually see at a glance.
    padded = (max(bbox[0] - 8, 0), max(bbox[1] - 8, 0), bbox[2] + 8, bbox[3] + 8)
    draw.rectangle(padded, outline=(255, 0, 0), width=4)

    output = BytesIO()
    annotated.save(output, format="PNG")
    return output.getvalue()

diff.getbbox() returns the smallest rectangle containing every non-zero pixel in the diff — i.e. the bounding box of all changed regions combined, not just one. For a page with multiple unrelated changes (a banner and a footer link both changed), this draws one box spanning both rather than two separate boxes. That's a real limitation worth knowing: if you need per-region boxes instead of one combined box, you'd need to find connected components in the diff mask rather than a single global bounding box — out of scope here, but worth flagging before you build an alerting pipeline around this that assumes one box means one change.

Noise reduction before drawing the box

A raw pixel diff still over-triggers on things like ad rotators, timestamps, or A/B-tested banners — annotating noise as if it were a real change defeats the purpose. Threshold the diff mask before computing the bounding box, same principle as the boolean check in Screenshot Monitoring:

def generate_diff_image(previous_bytes: bytes, current_bytes: bytes, noise_threshold: int = 24) -> bytes | None:
    prev = Image.open(BytesIO(previous_bytes)).convert("RGB")
    curr = Image.open(BytesIO(current_bytes)).convert("RGB")
    if prev.size != curr.size:
        curr = curr.resize(prev.size)

    diff = ImageChops.difference(prev, curr)
    # Zero out pixels below the noise threshold before computing the bbox,
    # so minor anti-aliasing/compression jitter doesn't produce a box.
    mask = diff.point(lambda p: 255 if p > noise_threshold else 0)
    bbox = mask.getbbox()
    if bbox is None:
        return None

    annotated = curr.copy()
    draw = ImageDraw.Draw(annotated)
    padded = (max(bbox[0] - 8, 0), max(bbox[1] - 8, 0), bbox[2] + 8, bbox[3] + 8)
    draw.rectangle(padded, outline=(255, 0, 0), width=4)
    output = BytesIO()
    annotated.save(output, format="PNG")
    return output.getvalue()

Attaching it to an alert

Slack, Discord, and most webhook receivers accept an image upload alongside a message. The exact upload mechanism depends on where the alert goes — the part specific to this pipeline is just that generate_diff_image() returns raw PNG bytes ready to attach:

import httpx

def send_diff_alert(webhook_url: str, page_url: str, diff_png: bytes) -> None:
    httpx.post(
        webhook_url,
        files={"file": ("diff.png", diff_png, "image/png")},
        data={"payload_json": f'{{"text": "Visual change detected on {page_url}"}}'},
        timeout=10,
    )

(Slack's incoming webhooks specifically don't accept file uploads this way — that example is illustrative of the general pattern; Slack requires the separate files.upload API with a bot token. Discord webhooks do accept multipart file uploads directly in this shape.)

Next steps

Screenshot Monitoring — the scheduled capture and threshold-comparison side this builds on
Website Change Detector — if you need to know a page's text content changed, not its visual appearance
Fetch API reference for screenshot capture options (device, wait_for)

Build a Hotel Price Drop Alert in n8n (No Polling Code Required)

XSron Hou — Mon, 21 Sep 2026 03:00:00 +0000

Originally posted on the Scrapio blog — sharing here too.

Rate-intelligence and rate-parity teams already know how to watch a hotel's price by hand: open Booking.com, check the same dates every morning, note what changed. Scrapio's hotel monitors automate the checking and the diffing — this guide connects that directly to n8n, so a price move lands in Slack, a spreadsheet, or wherever your workflow needs it, with zero polling code.

Unlike a typical n8n scraping workflow — where n8n's HTTP Request node polls Scrapio on a schedule — a monitor already owns the schedule and the diffing. n8n's job here is simpler: sit and wait for a webhook, then act on it.

What you'll need

n8n (cloud or self-hosted)
A Scrapio API key on a Starter plan or higher
A Booking.com property you want to track, plus fixed check-in/check-out dates and occupancy

Step 1 — Add a Webhook trigger in n8n

Create a new workflow, add a Webhook node as the trigger
Set HTTP Method to POST
Copy the Production URL n8n generates (something like https://your-instance.app.n8n.cloud/webhook/abc123)

This URL is where Scrapio will deliver monitor.change_detected events.

Step 2 — Register that URL as a Scrapio webhook endpoint

Webhook endpoints are managed from the dashboard, not a public API call: go to Dashboard → Webhooks, click Add Endpoint, paste the n8n URL from Step 1, and subscribe to the monitor.change_detected event. Scrapio hands back the endpoint's id (whe_...) and a signing secret — save the secret now, it's shown only once.

Step 3 — Create the hotel monitor, pointed at that endpoint

curl -X POST https://api.scrapio.dev/v1/booking/monitors \
  -H "Authorization: Bearer $SCRAPIO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Hotel Mansion — Amsterdam, Sep 10-12",
    "property_id": "nl/hotel-mansion",
    "check_in": "2026-09-10",
    "check_out": "2026-09-12",
    "adults": 2,
    "currency": "EUR",
    "cron": "0 * * * *",
    "watch": { "thresholds": { "abs": 5, "pct": 2 } },
    "webhook_endpoint_id": "whe_..."
  }'

Scrapio now checks this exact stay hourly. Nothing fires until the price actually moves past $5 or 2%, whichever comes first — see the Hotel Price Drop Alert template for the full parameter reference.

Step 4 — Parse the event in n8n

When a price changes, your Webhook node receives:

{
  "type": "monitor.change_detected",
  "data": {
    "schedule_id": "sch_...",
    "property_id": "nl/hotel-mansion",
    "params": { "check_in": "2026-09-10", "check_out": "2026-09-12", "adults": 2, "currency": "EUR" },
    "previous_price": 296,
    "current_price": 275,
    "currency": "EUR",
    "diff": -21,
    "detected_at": "2026-08-15T09:00:03Z"
  }
}

Add an IF node right after the Webhook trigger to branch on {{ $json.body.data.diff }} — route to your alert path only when diff < 0 (a genuine drop), and skip silently on a price increase.

Step 5 — Verify the signature (recommended)

Every delivery includes an X-Webhook-Signature header (t=<timestamp>,v1=<hex hmac>). Add a Code node before your IF branch that recomputes HMAC-SHA256(secret, "{timestamp}.{raw_body}") using the webhook endpoint's secret (returned once, at creation time, from Step 2) and compares it to the v1 value — reject anything that doesn't match before acting on it.

Step 6 — Send the alert

A complete 3-node workflow from here:

Webhook — receives monitor.change_detected
IF — only continues when diff < 0
Slack (or Gmail, or Google Sheets) — posts "{{ $json.body.data.property_id }} dropped from {{ $json.body.data.previous_price }} to {{ $json.body.data.current_price }} {{ $json.body.data.currency }}"

No cron node, no polling, no "last price" state to persist in n8n at all — Scrapio already owns that.

Batching into a daily digest instead

If you'd rather get one summary per day instead of a message per change, set "digest_interval_minutes": 1440 in the monitor's watch and subscribe your webhook endpoint to monitor.digest_delivered instead of (or alongside) monitor.change_detected — n8n receives one event per day per monitor, with every price move that happened batched into a changes array.

Next steps

See the Hotel Price Drop Alert template for the full API reference, including watching a specific point-of-sale market
Managing several properties? The Hotel Monitors dashboard lists all of them with live run history, no API calls required
For scraping workflows that aren't monitor-shaped (one-off lookups, batch jobs), see the general n8n web scraping workflow guide

Build an n8n Web Scraping Workflow Without Managing Infrastructure

XSron Hou — Mon, 14 Sep 2026 03:00:00 +0000

Originally posted on the Scrapio blog — sharing here too.

n8n is one of the best workflow automation tools available — but its built-in HTTP nodes hit a wall fast when you're scraping real sites. JavaScript-rendered pages fail silently. Sites block datacenter IPs. Sessions don't persist.

Scrapio handles all of that behind a single API call, making it a natural backend for n8n scraping workflows.

What you'll need

n8n (cloud or self-hosted)
A Scrapio API key

Step 1 — Store your API key in n8n credentials

In n8n, go to Settings → Credentials → New Credential
Choose HTTP Request (generic)
Set Authentication to Header Auth
Name: Authorization, Value: Bearer sk-YOUR_KEY
Save as Scrapio API

Step 2 — Create the workflow

Add an HTTP Request node with these settings:

Field	Value
Method	POST
URL	`https://api.scrapio.dev/v1/fetch`
Authentication	Scrapio API (from step 1)
Body Content Type	JSON

Body:

{
  "url": "={{ $json.url }}",
  "output": ["markdown"]
}

The ={{ $json.url }} expression pulls the URL from whatever node feeds into this one — a Schedule trigger, a spreadsheet, a webhook, etc.

Step 3 — Parse the output

The response looks like:

{
  "request_id": "req_abc123",
  "mode": "inline",
  "status": "completed",
  "outputs": {
    "markdown": "# Page Title\n\nContent here..."
  },
  "usage": { "credits": 1 }
}

Add a Set node to extract {{ $json.outputs.markdown }} and pass it to downstream nodes — a Google Sheet, an AI summarizer, a database insert, whatever your workflow needs.

Example: daily competitor price monitor

A complete 4-node workflow:

Schedule Trigger — runs every day at 8am
HTTP Request (Scrapio) — fetches competitor pricing page as markdown
OpenAI node — extracts prices with a prompt: "From this markdown, return a JSON object with product names as keys and prices as values"
Google Sheets — appends the result to a tracking sheet

No servers. No Playwright. No proxy rotation. The entire workflow runs in n8n cloud and costs pennies per day.

Handling JavaScript-rendered pages

Add "render_js": true to the request body for SPAs and dynamic pages:

{
  "url": "={{ $json.url }}",
  "output": ["markdown"],
  "render_js": true
}

Handling multiple URLs

Feed a list of URLs from a Spreadsheet File or Airtable node, split with SplitInBatches, and pass each through the Scrapio HTTP Request node. n8n handles the loop; Scrapio handles the rendering.

Next steps

See the Webpage to Markdown template for a copy-paste API recipe
Read about scheduled scraping jobs for cron-based automation
Check the Scrapio Fetch docs for all request options

Crawl a Competitor Site and Export to JSON

XSron Hou — Mon, 07 Sep 2026 03:00:00 +0000

Originally posted on the Scrapio blog — sharing here too.

Understanding a competitor's content strategy means knowing what pages they have, what topics they cover, and how they structure their information. Manual browsing doesn't scale. Scrapio's Crawl endpoint spiders an entire domain automatically and returns everything as structured data.

What you'll need

A Scrapio API key (any plan — Crawl isn't gated to a specific tier)
Python

Start a crawl

Pass seeds (an array of starting URLs), max_pages (up to 50), max_depth, and an optional extract object to pull structured data from each page.

import httpx
import json

API_KEY = "sk-..."
BASE = "https://api.scrapio.dev"

resp = httpx.post(
    f"{BASE}/v1/crawl",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "seeds": ["https://competitor.com"],
        "max_pages": 50,
        "max_depth": 4,
        "same_domain_only": True,
        "output": ["json"],
        "extract": {
            "mode": "schema",
            "schema": {
                "title": "string",
                "description": "string",
                "word_count": "integer",
                "publish_date": "string",
            },
        },
    },
    timeout=300,
)
resp.raise_for_status()
result = resp.json()

The crawl runs synchronously and returns when complete.

Inspect the results

pages = result["result"]["pages"]
summary = result["result"]["summary"]

print(f"Pages fetched:    {summary['pages_fetched']}")
print(f"Pages succeeded:  {summary['pages_succeeded']}")
print(f"Pages failed:     {summary['pages_failed']}")

Each page in pages looks like:

{
  "url": "https://competitor.com/blog/web-scraping-guide",
  "depth": 1,
  "status": "completed",
  "outputs": {
    "json": {
      "title": "The Complete Guide to Web Scraping",
      "description": "Everything you need to know about extracting data from websites.",
      "word_count": 3842,
      "publish_date": "2026-03-15"
    }
  }
}

Export to JSON

with open("competitor_content.json", "w") as f:
    json.dump(pages, f, indent=2)

print(f"Exported {len(pages)} pages")

Analyze with pandas

import pandas as pd

rows = [
    {
        "url": p["url"],
        "depth": p["depth"],
        "status": p["status"],
        **p.get("outputs", {}).get("json", {}),
    }
    for p in pages
    if p["status"] == "completed"
]
df = pd.DataFrame(rows)

print("Average word count:", df["word_count"].mean())
print("\nTop 10 longest pages:")
print(df.nlargest(10, "word_count")[["url", "word_count"]])

What to look for

Content gaps — topics they cover that you don't
Word count distribution — how long are their top-performing pieces?
Update frequency — filter by publish_date to see how often they publish
URL structure — how is their site organized? What categories do they prioritize?

Next steps

Try the Full Site Crawl to JSON template for the complete recipe
See Site Map Discovery to map a domain's URL structure without downloading full content
Read the Crawl API docs

Schedule a Weekly Price Monitor in 10 Minutes

XSron Hou — Mon, 31 Aug 2026 03:00:00 +0000

Originally posted on the Scrapio blog — sharing here too.

Price monitoring is a common automation need — whether you're tracking a competitor, waiting for a deal, or monitoring supplier costs. The traditional path involves a server, a cron job, a scraper, and a notification system. With Scrapio you only need the last two.

Scrapio now also has native Monitors — set the URL, the fields to extract, and which ones to watch from the dashboard, and Scrapio owns the schedule, the "last value" comparison, and the webhook delivery for you. The walkthrough below is still useful if you want the cron trigger and comparison logic running in your own infrastructure (e.g. to chain into other systems), but for most teams the native Monitor is the faster path — skip to Next steps to try it.

What you'll need

A Scrapio API key
A cron trigger (GitHub Actions, cron-job.org, or any scheduler)
A webhook URL to receive notifications (use webhook.site for testing)

The approach

The pattern is simple:

A cron trigger fires on a schedule (e.g. every Monday at 9am)
It calls Scrapio to extract the current price from the product page
It compares against the last recorded price
If the price dropped, it sends a notification

Step 1 — Fetch the price with Scrapio

Use the extract parameter to pull just the fields you care about:

import httpx

API_KEY = "sk-..."

def get_price(url: str) -> dict:
    resp = httpx.post(
        "https://api.scrapio.dev/v1/fetch",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={
            "url": url,
            "output": ["json"],
            "extract": {
                "mode": "schema",
                "schema": {
                    "price": "number",
                    "availability": "string",
                },
            },
            "render_js": True,
        },
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json()["outputs"]["json"]

Step 2 — Compare and notify

import json
import os
import httpx

PRICE_FILE = "last_price.json"
WEBHOOK_URL = "https://your-webhook-url.com/price-alert"
PRODUCT_URL = "https://example.com/products/iphone-15"

def load_last_price() -> float | None:
    if os.path.exists(PRICE_FILE):
        return json.loads(open(PRICE_FILE).read()).get("price")
    return None

def save_price(price: float) -> None:
    open(PRICE_FILE, "w").write(json.dumps({"price": price}))

def run():
    current = get_price(PRODUCT_URL)
    price = current["price"]
    last = load_last_price()

    if last is not None and price < last:
        httpx.post(WEBHOOK_URL, json={
            "message": f"Price dropped from ${last:.2f} to ${price:.2f}",
            "url": PRODUCT_URL,
            "price": price,
        })

    save_price(price)

run()

Step 3 — Schedule with GitHub Actions

Create .github/workflows/price-monitor.yml:

name: Price Monitor
on:
  schedule:
    - cron: "0 9 * * 1"  # Every Monday at 9am UTC
  workflow_dispatch:

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install httpx
      - run: python monitor.py
        env:
          SCRAPIO_API_KEY: ${{ secrets.SCRAPIO_API_KEY }}

GitHub Actions is free for public repos and the free tier covers hundreds of runs per month for private ones. No server required.

Cron expressions quick reference

Expression	Meaning
`0 9 * * 1`	Every Monday at 9am UTC
`0 /6 * *`	Every 6 hours
`0 8 * * *`	Every day at 8am UTC
`/30 * * *`	Every 30 minutes

Next steps

Try the native Monitors dashboard — no cron trigger or webhook server to host
See the Native Price Monitoring template for a step-by-step walkthrough
Monitoring content beyond price? See Website Change Detector — or use a native monitor with "watch": {"mode": "content"} for the same thing without hosting anything
Prefer to run it yourself? Try the Price Drop Alert template for the GitHub Actions recipe
Read the Fetch API docs

Take Screenshots of Any Website via API

XSron Hou — Mon, 31 Aug 2026 03:00:00 +0000

Originally posted on the Scrapio blog — sharing here too.

Screenshots via API have more use cases than you might expect: visual regression testing, monitoring landing pages for unauthorized changes, generating OG image previews, archiving pages, and building internal dashboards with live page previews.

What you'll need

A Scrapio API key
curl or Python

Take a basic screenshot

curl -X POST https://api.scrapio.dev/v1/fetch \
  -H "Authorization: Bearer sk-..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "output": ["screenshot"],
    "render_js": true
  }'

Screenshot output always requires render_js: true — without it, the request is routed to a non-browser executor that can't capture a screenshot and the call fails.

The response contains a screenshot object with a url field pointing to the captured image:

{
  "request_id": "req_abc123",
  "mode": "inline",
  "status": "completed",
  "outputs": {
    "screenshot": {
      "url": "https://storage.scrapio.dev/screenshots/req_abc123.png"
    }
  },
  "usage": { "credits": 2 }
}

Capture a mobile viewport

Pass device: "mobile" to render the page at a mobile viewport:

import httpx

resp = httpx.post(
    "https://api.scrapio.dev/v1/fetch",
    headers={"Authorization": "Bearer sk-..."},
    json={
        "url": "https://example.com",
        "output": ["screenshot"],
        "render_js": True,
        "device": "mobile",
    },
    timeout=30,
)
screenshot_url = resp.json()["outputs"]["screenshot"]["url"]
print(screenshot_url)

device accepts "desktop" (default), "mobile", or "tablet".

Wait for JavaScript content to finish loading

render_js: true is required for every screenshot request. For pages that load content asynchronously, pair it with wait_for:

resp = httpx.post(
    "https://api.scrapio.dev/v1/fetch",
    headers={"Authorization": "Bearer sk-..."},
    json={
        "url": "https://example.com/dashboard",
        "output": ["screenshot"],
        "render_js": True,
        "wait_for": {"network_idle": True},
    },
    timeout=30,
)
screenshot_url = resp.json()["outputs"]["screenshot"]["url"]

wait_for: {network_idle: true} tells Scrapio to hold until all network requests settle before capturing — useful for charts and dashboards that load data asynchronously.

Combine screenshot with markdown

You can request multiple outputs in a single call:

resp = httpx.post(
    "https://api.scrapio.dev/v1/fetch",
    headers={"Authorization": "Bearer sk-..."},
    json={
        "url": "https://example.com/article",
        "output": ["screenshot", "markdown"],
        "render_js": True,
    },
    timeout=30,
)
outputs = resp.json()["outputs"]
screenshot_url = outputs["screenshot"]["url"]
text = outputs["markdown"]

Visual monitoring workflow

Download the screenshot and diff against a baseline using Pillow:

import httpx
from PIL import Image, ImageChops
import io

def screenshot_url(url: str) -> Image.Image:
    resp = httpx.post(
        "https://api.scrapio.dev/v1/fetch",
        headers={"Authorization": "Bearer sk-..."},
        json={"url": url, "output": ["screenshot"], "render_js": True},
        timeout=30,
    )
    img_url = resp.json()["outputs"]["screenshot"]["url"]
    img_bytes = httpx.get(img_url).content
    return Image.open(io.BytesIO(img_bytes))

def has_visual_change(baseline: Image.Image, current: Image.Image) -> bool:
    diff = ImageChops.difference(baseline, current)
    return diff.getbbox() is not None

Next steps

Read the Fetch API docs for all options
See Website Change Detector for content-based monitoring

How to Scrape Google Search Results with an API

XSron Hou — Mon, 24 Aug 2026 03:01:00 +0000

Originally posted on the Scrapio blog — sharing here too.

Google Search is one of the most valuable data sources on the internet, and one of the hardest to scrape reliably. Google detects and blocks bots aggressively — rate limits, CAPTCHAs, and IP bans make direct scraping fragile.

Scrapio routes search requests through a specialized proxy layer that mirrors real browser behavior. You get clean, structured results without any of the blocking.

What you'll need

A Scrapio API key on a Pro plan or higher (Google Search isn't available on Free or Starter)
curl or Python

Basic search request

The Google Search endpoint is a GET request — pass parameters as query strings:

curl "https://api.scrapio.dev/v1/google/search?search=best+web+scraping+api+2026" \
  -H "Authorization: Bearer sk-..."

Response:

{
  "organic_results": [
    {
      "position": 1,
      "title": "Top 10 Web Scraping APIs Compared (2026)",
      "url": "https://example.com/web-scraping-apis",
      "description": "We tested 10 web scraping APIs on speed, reliability, and price...",
      "domain": "example.com"
    }
  ],
  "questions": [
    "Is web scraping legal?",
    "What is the difference between scraping and crawling?",
    "How do I scrape Google without getting blocked?"
  ],
  "related_searches": [...]
}

Paginate results

import httpx

API_KEY = "sk-..."

def get_serp_page(query: str, page: int = 1) -> dict:
    return httpx.get(
        "https://api.scrapio.dev/v1/google/search",
        headers={"Authorization": f"Bearer {API_KEY}"},
        params={
            "search": query,
            "page": page,
        },
        timeout=30,
    ).json()

# Collect first 3 pages of results
all_organic = []
for page in range(1, 4):
    results = get_serp_page("web scraping api", page)
    all_organic.extend(results.get("organic_results", []))

print(f"Collected {len(all_organic)} results")

Localize results

Pass country_code (ISO 3166-1 alpha-2) and language to get localized SERPs:

httpx.get(
    "https://api.scrapio.dev/v1/google/search",
    headers={"Authorization": f"Bearer {API_KEY}"},
    params={
        "search": "web scraping api",
        "country_code": "de",
        "language": "de",
    },
    timeout=30,
).json()

Search type

Use search_type to query different Google verticals:

Value	Description
`classic`	Standard web search (default)
`news`	Google News results
`images`	Google Images
`lens`	Google Lens (visual search)
`ai_mode`	Google AI Mode results
`ads`	Google Ads results

shopping and maps aren't supported — Google blocks them on every provider path this API has access to, so they're rejected at validation rather than silently failing.

params={"search": "wireless headphones", "search_type": "images"}

Use cases

SEO rank tracking — monitor your keyword positions daily
Competitor research — see what's ranking for your target keywords
Market research — extract PAA boxes to understand user intent
Content strategy — find gaps in top-ranking pages

Next steps

Try the Google SERP to JSON template for a ready-to-run recipe
See Google News Monitor for tracking news coverage by keyword
Read the Google Search API docs

How to Extract Structured JSON from Any Website

XSron Hou — Mon, 17 Aug 2026 03:00:00 +0000

Originally posted on the Scrapio blog — sharing here too.

Every data pipeline eventually hits the same wall: the information you need is on a web page, but it comes back as a blob of HTML. Writing CSS selectors works until the site redesigns. Regex works until it doesn't.

Scrapio's extract parameter takes a different approach — you describe the shape of the data you want, and the API figures out how to pull it from any page.

What you'll need

A Scrapio API key
The URL of a page with structured content (product page, article, listing)
curl or Python

Define a schema

Pass output: ["json"] and an extract object with mode: "schema". The schema maps field names to their types — the API handles figuring out where those values live on the page.

curl -X POST https://api.scrapio.dev/v1/fetch \
  -H "Authorization: Bearer sk-..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products/wireless-headphones",
    "output": ["json"],
    "extract": {
      "mode": "schema",
      "schema": {
        "name": "string",
        "price": "number",
        "currency": "string",
        "availability": "string",
        "rating": "number",
        "review_count": "integer"
      }
    }
  }'

Response:

{
  "request_id": "req_abc123",
  "mode": "inline",
  "status": "completed",
  "outputs": {
    "json": {
      "name": "Sony WH-1000XM5 Wireless Headphones",
      "price": 279.99,
      "currency": "USD",
      "availability": "In Stock",
      "rating": 4.7,
      "review_count": 12483
    }
  },
  "usage": { "credits": 5 }
}

Schema and instruction extraction run through the LLM extractor by default (5 credits) whenever an LLM provider is configured on your account; it only falls back to the cheaper heuristic extractor (2 credits) if no provider is configured or the LLM call fails. Set "engine": "heuristic" in the extract object if you want to guarantee the 2-credit rate.

Extract with a natural-language instruction

For more complex extractions, use mode: "instruction" to describe what you want in plain English:

import httpx

resp = httpx.post(
    "https://api.scrapio.dev/v1/fetch",
    headers={"Authorization": "Bearer sk-..."},
    json={
        "url": "https://example.com/search?q=headphones",
        "output": ["json"],
        "extract": {
            "mode": "instruction",
            "instruction": "Extract all product listings. For each product return: title, price, URL, and rating.",
        },
    },
    timeout=30,
)
items = resp.json()["outputs"]["json"]
print(f"Found {len(items)} products")

Extract using CSS selectors

When you know the page structure, mode: "selectors" is the most precise option:

resp = httpx.post(
    "https://api.scrapio.dev/v1/fetch",
    headers={"Authorization": "Bearer sk-..."},
    json={
        "url": "https://example.com/products/headphones",
        "output": ["json"],
        "extract": {
            "mode": "selectors",
            "fields": {
                "title":  {"selector": "h1.product-title", "type": "text"},
                "price":  {"selector": ".price-now",       "type": "text"},
                "rating": {"selector": "[data-rating]",    "type": "attr", "attribute": "data-rating"},
            },
        },
    },
    timeout=30,
)
data = resp.json()["outputs"]["json"]

Tips

Keep schemas focused. Only request fields you need — the more fields, the harder the extraction.
Use render_js: true for pages that load prices or content via JavaScript.
Validate downstream. For production pipelines, validate the output shape before inserting into a database.

Next steps

Try the Product Price Extractor template
Read the Fetch API docs
See Full Site Crawl to JSON for bulk extraction across an entire domain

How to Feed Live Website Data into an LLM Agent

XSron Hou — Mon, 10 Aug 2026 03:00:00 +0000

Originally posted on the Scrapio blog — sharing here too.

LLM agents are only as good as the context you give them. Static knowledge cuts off at a training date — but most interesting tasks require knowing what a page says right now: a competitor's pricing, a news article, a product listing.

The bottleneck is usually getting clean text into the prompt. Raw HTML is noisy. Playwright is complex. Here's the simple path.

What you'll need

A Scrapio API key (free tier covers this example)
An LLM with a large enough context window (GPT-4o, Claude 3.5, Gemini 1.5 work well)
Python 3.10+

Fetch a page as markdown

Scrapio returns clean markdown by default. One API call strips navigation, ads, and boilerplate — leaving just the readable content.

import httpx

API_KEY = "sk-..."

def get_page_markdown(url: str) -> str:
    resp = httpx.post(
        "https://api.scrapio.dev/v1/fetch",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"url": url, "output": ["markdown"]},
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json()["outputs"]["markdown"]

Pass it to your agent

from openai import OpenAI

client = OpenAI()

def answer_about_page(url: str, question: str) -> str:
    page_content = get_page_markdown(url)

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": "You are a research assistant. Answer based only on the provided web page content.",
            },
            {
                "role": "user",
                "content": f"Page content:\n\n{page_content}\n\nQuestion: {question}",
            },
        ],
    )
    return response.choices[0].message.content

print(answer_about_page(
    "https://example.com/pricing",
    "What is the cheapest paid plan and what does it include?"
))

Why markdown beats raw HTML

Format	Tokens per page (avg)	Noise
Raw HTML	8,000–40,000	High (tags, scripts, styles)
Markdown	800–4,000	Low

Markdown is 5–10× smaller, which means lower cost and less context pressure on your model.

Handling JavaScript-heavy pages

If the target page loads content via JavaScript (SPAs, dashboards), set render_js to true:

json={"url": url, "output": ["markdown"], "render_js": True}

Scrapio spins up a headless browser, waits for the page to settle, and returns the rendered content.

Next steps

Try the Webpage to Markdown template for a ready-to-run example
Read the Fetch API docs for all available options
See RAG pipelines with crawled sites for bulk ingestion

Why Markdown Is Better Than HTML for LLM Context Windows

XSron Hou — Wed, 05 Aug 2026 15:43:09 +0000

Originally posted on the Scrapio blog — sharing here too.

When you're building a RAG pipeline or an LLM agent that reads web pages, the format you put into the context window matters more than most people realize. Raw HTML and clean markdown can represent the same information, but one costs 10× more in tokens and adds substantial noise that degrades model performance.

The problem with HTML

A typical product page might have 3,000 words of actual content. As raw HTML, that page is often 50,000–80,000 characters — mostly tags, class names, JavaScript, tracking pixels, and navigation menus.

LLMs tokenize all of it. You pay for every <div class="product-carousel__item--highlighted"> even though it carries no information your model needs.

Measured comparison

We ran 100 pages through both approaches and measured:

Metric	Raw HTML	Clean Markdown
Avg tokens per page	18,400	1,850
Tokens that are content	~12%	~94%
GPT-4o cost per 1,000 pages	~$91	~$9
Extraction accuracy (structured tasks)	71%	89%

The accuracy difference is real: LLMs perform better when they're not trying to reason through a wall of HTML noise.

What clean markdown looks like

Raw HTML snippet:

<div class="pdp-price-container" data-testid="price-wrapper">
  <span class="price--sale" aria-label="Sale price">
    <span class="price__currency">$</span>
    <span class="price__amount">279</span>
    <span class="price__cents">.99</span>
  </span>
</div>

Same content as markdown:

**Price:** $279.99

The markdown version is 22 tokens. The HTML version is 89 tokens — and that's a small excerpt. An entire page compounds this by 50–100×.

Getting clean markdown from any URL

curl -X POST https://api.scrapio.dev/v1/fetch \
  -H "Authorization: Bearer sk-..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/article", "output": ["markdown"]}'

Response:

{
  "request_id": "req_abc123",
  "mode": "inline",
  "status": "completed",
  "outputs": {
    "markdown": "# Article Title\n\nContent here..."
  },
  "usage": { "credits": 1 }
}

The markdown output strips navigation, ads, scripts, and layout elements. What remains is the readable content — headings, paragraphs, lists, tables, and code blocks.

JavaScript-rendered pages

SPAs and dynamic pages need a render step:

{
  "url": "https://example.com/dashboard",
  "output": ["markdown"],
  "render_js": true
}

Without render_js, you'd get the pre-hydration HTML — often an empty shell.

For RAG pipelines

When crawling large sites for a vector database, markdown is especially important because:

Chunking is cleaner — markdown structure (headings, paragraphs) provides natural split points
Embeddings are better — semantic models trained on clean text produce better vectors when the input isn't polluted with HTML syntax
Storage is cheaper — 10× smaller documents mean lower storage and fewer API calls to retrieve them

Next steps

Try the Webpage to Markdown template for a copy-paste implementation
See Batch Article Extractor for crawling and indexing entire sites
Read How to Feed Live Website Data into an LLM Agent