Website content crawler: how to get clean page text as JSON or CSV (Python, API, no code)

Crawls any public website from a start URL, follows internal links up to a set depth, and extracts title, breadcrumbs, clean text body, and word count.

This tutorial shows what the Actor v0iddo/website-content-crawler returns, how to run it on the Apify Store without code, and how to call it from Python and cURL.

What data you get

One row per result. Every row can be downloaded as JSON, CSV, Excel or HTML, or read through the API.

Field	Label	Type	Example
url	Page URL	URL	https://flask.palletsprojects.com/en/stable/
title	Title	text	Welcome to Flask
h1	H1 Heading	text	Welcome to Flask
breadcrumbs	Breadcrumbs	list	["Flask Documentation (3.1.x)", "Welcome to Flask"]
description	Description	text
textContent	Text content	text	Welcome to Flask Welcome to Flask’s documentation. Flask is a ligh...
wordCount	Word count	number	920
depth	Crawl depth	number	0

Real rows from a run of the Actor (fields that name people are left out):

[
  {
    "url": "https://flask.palletsprojects.com/en/stable/",
    "title": "Welcome to Flask",
    "h1": "Welcome to Flask",
    "breadcrumbs": [
      "Flask Documentation (3.1.x)",
      "Welcome to Flask"
    ],
    "textContent": "Welcome to Flask  Welcome to Flask’s documentation. Flask is a lightweight WSGI web application framework.\nIt is designed to make getting...",
    "wordCount": 920,
    "depth": 0
  },
  {
    "url": "https://flask.palletsprojects.com/en/stable/genindex/",
    "title": "Index",
    "h1": "Index",
    "breadcrumbs": [
      "Flask Documentation (3.1.x)",
      "Index"
    ],
    "textContent": "Index _ | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y _ _AppCtxGlobals (class in flask....",
    "wordCount": 1644,
    "depth": 1
  }
]

No code: run it on the Apify Store

1. Open the Website content crawler page on Apify.
2. Fill in the input form. An example input is already filled in.
3. Press Start, wait for the run to finish, and download the results as JSON, CSV or Excel from the Output tab.

The input that is pre-filled on the Actor page:

{
  "startUrls": [
    "https://docs.python.org/3/library/"
  ],
  "maxDepth": 1,
  "maxPagesPerUrl": 15,
  "maxItems": 20
}

Python (apify-client)

Get your API token in Apify Console under Settings, then Integrations. The Actor id is v0iddo/website-content-crawler.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run_input = {'startUrls': ['https://docs.python.org/3/library/'],
 'maxDepth': 1,
 'maxPagesPerUrl': 15,
 'maxItems': 20}

run = client.actor("v0iddo/website-content-crawler").call(run_input=run_input)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

cURL

curl -X POST "https://api.apify.com/v2/acts/v0iddo~website-content-crawler/run-sync-get-dataset-items" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://docs.python.org/3/library/"], "maxDepth": 1, "maxPagesPerUrl": 15, "maxItems": 20}'

Good to know

• The crawler stays within the same origin and path prefix as each start URL. Use includeUrlPattern or excludeUrlPattern to narrow scope on large sites; combine with maxPagesPerUrl to control cost.
• Set maxDepth=0 to extract only the start page. maxDepth=1 adds all pages linked from the start page. Depths above 3 can produce thousands of pages on large sites.
• Pages behind login, paywalls, or bot walls are not accessible.
• Not affiliated with any documentation platform or website.

Price

$2.55 per 1,000 results. You pay per result row. There is no separate charge per run. Higher Apify plans pay less per result, down to $1.78 per 1,000.

FAQ

What data does the Website content crawler return?

Each result is one row in an Apify dataset. Fields include url, title, h1, breadcrumbs, description, textContent, wordCount, depth. The table above shows every field with an example value.

Can I use it without writing code?

Yes. Open the Actor page on Apify, fill in the input form (an example is already filled in), press Start, and download the results as JSON, CSV, Excel or HTML. You can also schedule runs there.

How much does it cost?

$2.55 per 1,000 results. You pay per result row. There is no separate charge per run. Higher Apify plans pay less per result, down to $1.78 per 1,000. A new Apify account gets $5 of platform credit every month.

Links

• Actor on the Apify Store: Website content crawler
• Full guide with every field and the JavaScript example: https://voidd0.github.io/website-content-crawler/