Crawlee is most useful when your task has moved beyond “fetch one URL.” It gives you a crawl loop with a request queue, handlers, routing, and dataset storage, while you write the page-specific extraction logic.
This tutorial uses the Books to Scrape demonstration catalog. It is intentionally bounded to three requests.
Requirements and install
The current Crawlee Python quick start lists version 1.11 and requires Python 3.10 or newer. Follow the official install command there and pin the version you validate in your project.
Build the crawler
import asyncio
from crawlee.crawlers import (
BeautifulSoupCrawler,
BeautifulSoupCrawlingContext,
)
async def main() -> None:
# Keep the first run small so you can inspect scope and output.
crawler = BeautifulSoupCrawler(max_requests_per_crawl=3)
@crawler.router.default_handler
async def handle_page(context: BeautifulSoupCrawlingContext) -> None:
soup = context.soup
for card in soup.select("article.product_pod"):
title_link = card.select_one("h3 a")
price = card.select_one(".price_color")
availability = card.select_one(".availability")
if title_link is None:
continue
await context.push_data(
{
"source_url": context.request.url,
"title": title_link.get("title")
or title_link.get_text(strip=True),
"product_url": title_link.get("href"),
"price_text": price.get_text(strip=True)
if price
else None,
"availability_text": availability.get_text(" ", strip=True)
if availability
else None,
}
)
# Follow only the documented next-page link in this demo catalog.
await context.enqueue_links(selector="li.next a")
await crawler.run(["https://books.toscrape.com/"])
await crawler.export_data(path="books.csv")
if __name__ == "__main__":
asyncio.run(main())
Run it:
python books_catalog.py
The exported dataset contains one record per product card, with a source URL that lets you trace the record back to the page. For a production pipeline, normalize relative product links with Python’s urljoin, add a crawl timestamp, and validate required fields before accepting a row.
What Crawlee manages
- Queue: schedules discovered requests.
- Handler: defines extraction for a page.
- Dataset: stores structured records produced by handlers.
- Request cap: bounds this run to a known number of pages.
The handler remains your code. You decide which selectors matter, how to normalize values, and what constitutes a valid record.
HTTP crawler or PlaywrightCrawler?
Start with BeautifulSoupCrawler or ParselCrawler when the response HTML already contains the content. The HTTP crawlers do not render JavaScript. Use PlaywrightCrawler when an authorized target loads required content in the browser after JavaScript execution or interaction.
A browser crawler has more runtime and resource overhead. Test the same representative URLs with both approaches and compare:
- Presence of required fields
- Missing or duplicate records
- Median and tail latency
- Browser worker memory and throughput
- Failure behavior and retry visibility
Do not count a successful navigation as a successful extraction. Assert that the selector or content you need exists.
Add boundaries before scaling
For your own crawler, configure explicit:
- Allowed hostnames and URL patterns
- Request or page caps
- Maximum depth where relevant
- Concurrency and request pacing
- Timeout and retry policy
- Record validation and deduplication
The demo follows one next-page link. A real production crawl should also avoid leaving the intended domain and should respect the target’s access rules. Queue deduplication is useful, but it does not replace URL normalization or an explicit scope.
When Crawlee is a fit
Use Crawlee when you want the framework to organize a custom Python crawling workflow while keeping extraction code under your control. Scrapy may fit teams that want its ecosystem and deeper downloader pipeline control. Playwright alone is better when browser automation is the task and you do not need a full crawler workflow.
If you would rather not run crawler workers, a managed API is another operating model. It can reduce infrastructure work but gives you a different level of control over scheduling and extraction. Evaluate on your own pages and acceptance criteria.
Discussion: Do you start your crawlers with an HTTP client and add browser rendering only after seeing missing content, or do your targets require a browser from day one?
References: Crawlee Python docs · Scrapy spiders · Playwright Python docs
Top comments (1)
The three-request cap makes this a useful fixture. I would also assert that every exported product_url is absolute and unique before increasing it. BrowserAct Cloud offers a hosted reusable-Bot option for the same extraction job, but the source URL, crawl timestamp and required-field checks still belong in the output contract.