Scrape any website into JSON by just listing the fields you want

Tidy Tools — Mon, 28 Sep 2026 20:53:48 +0000

Writing a scraper usually means inspecting the page, finding CSS selectors, handling edge cases, and then fixing it all again when the site changes its layout. For a lot of jobs that's overkill: you just want these five facts from these pages.

So I built an extractor that works the other way around. You describe the data, an LLM reads the page, and you get JSON back.

The whole input

{
  "startUrls": [{ "url": "https://github.com/apify/crawlee" }],
  "fields": {
    "name": "string",
    "description": "string",
    "license": "string",
    "primary_language": "string"
  }
}

The output (a real run)

{
  "url": "https://github.com/apify/crawlee",
  "success": true,
  "data": {
    "name": "crawlee",
    "description": "Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers...",
    "license": "Apache License 2.0",
    "primary_language": "JavaScript"
  }
}

No selectors, no code, and the same input works on any other repository page, or on a completely different site if you change the fields.

Things I cared about

No invented data. The model is instructed to use only what's on the page and return null otherwise. Missing is better than made up.
Types you choose. string, number, integer, boolean or array per field, or a full JSON Schema for nested results like "all jobs on this page with title, location and remote flag".
Plain-language instructions. "Prices in USD as numbers" or "only the first listing" steer the result without code.
Works on stubborn sites. Pages are fetched over plain HTTP first; if a site blocks that or needs JavaScript, a real browser takes over automatically.
Pay only for results. Blocked, failed and empty pages are free. It's $0.01 per extracted page, about a third of what comparable AI scrapers on the same platform charge.

Good fits

Product pages → name, price, availability
Job posts → title, company, location, salary, remote
Company websites → what they do, industry, headquarters, contact page
Articles → headline, author, date, summary
Event pages → date, venue, price

When not to use it

If you need millions of pages from one site with a fixed layout, a classic selector-based scraper is cheaper. This tool shines when pages vary, layouts change, or you only need a handful of fields from many different sites.

Try it

AI Web Data Extractor on Apify. New Apify accounts get free monthly credit, so you can try it on your own URLs for free. It can also be called from code or used as a tool by AI agents through the Apify MCP server.

Feedback welcome, especially examples where it gets something wrong.

Turn any docs site into RAG-ready Markdown for $1 per 1,000 pages

Tidy Tools — Mon, 28 Sep 2026 20:28:05 +0000

Every RAG project starts the same way: "first, get the docs into clean text." And every time it takes longer than it should: menus and cookie banners leak into your chunks, JavaScript-rendered pages come back empty, PDFs need a separate pipeline, and crawling the same page under five URLs quietly inflates your embedding bill.

I built a small crawler that handles all of that and runs on Apify, so you can call it from code, a schedule, or an AI agent.

What it does

Give it a start URL and a page limit:

{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "maxPages": 200,
  "cssSelector": "main",
  "chunkSize": 2000
}

You get one item per page:

{
  "url": "https://docs.example.com/getting-started",
  "title": "Getting started",
  "markdown": "# Getting started\n\nInstall the package...",
  "chunks": [{ "index": 0, "text": "# Getting started..." }],
  "mode": "fast"
}

The details that matter for RAG

Stays in scope. By default it only follows links under the start URL's folder, so /docs/ doesn't wander into /blog/ and /careers/.
Clean content. A CSS selector like main drops navigation and footers before conversion.
Documents included. Linked PDFs, Word and Excel files are converted to Markdown in the same run.
Chunking built in. Paragraph-aware chunks with overlap, ready for your embedding model.
Duplicates removed. Pages with identical content under different URLs (tracking params, session ids) are stored once and not charged twice.
Fast first, browser when needed. Most pages are fetched over plain HTTP (fast, $1 / 1,000 pages). If a site blocks that or is a JavaScript shell, it switches to a real headless browser automatically ($2.50 / 1,000 pages).
Pay only for success. Blocked pages, errors and duplicates are free.

Calling it from Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("tidytools/website-markdown-crawler").call(run_input={
    "startUrls": [{"url": "https://docs.example.com/"}],
    "maxPages": 100,
    "chunkSize": 2000,
})
for page in client.dataset(run["defaultDatasetId"]).iterate_items():
    for chunk in page.get("chunks", []):
        index.add(text=chunk["text"], metadata={"url": page["url"], "title": page["title"]})

From an AI agent

Because it's an Apify Actor, agents can use it as a tool through the Apify MCP server: "read the docs at X and answer Y" becomes one tool call.

Try it

Website to Markdown Crawler on Apify. New Apify accounts get free monthly credit, so a few hundred pages cost nothing to try.

If you need structured data instead of Markdown (prices, job details, company info), there's a sister tool, AI Web Data Extractor: list the fields you want and get JSON back.

Feedback and feature requests are very welcome.

DEV Community: Tidy Tools

Scrape any website into JSON by just listing the fields you want

The whole input

The output (a real run)

Things I cared about

Good fits

When not to use it

Try it

Turn any docs site into RAG-ready Markdown for $1 per 1,000 pages

What it does

The details that matter for RAG

Calling it from Python

From an AI agent

Try it