DEV Community: Zyte

Stop Scraping HTML - There's a better way.

John Rooney — Tue, 16 Dec 2025 19:16:43 +0000

The "API-First" Reverse Engineering Method

One of the most common mistakes I see developers make is firing up their code editor too early. They open VS Code, pip install requests beautifulsoup4, and immediately start trying to parse <div> tags.

If you are scraping a modern e-commerce site or Single Page Application (SPA), this is the wrong approach. It’s brittle, it’s slow, and it breaks the moment the site updates its CSS.

The secret to scalable scraping isn't better parsing; it's finding the API that the website uses to populate itself. Here is the exact workflow I use to turn a complex parsing job into a clean, reliable JSON pipeline.

Phase 1: The Discovery (XHR Filtering)

Modern websites are rarely static. They typically use a "Frontend/Backend" architecture where the browser loads a skeleton page and then fetches the actual data via a background API call.

Your goal is to use that call.

Open Developer Tools: Right-click and inspect the page, then navigate to the Network tab.
Filter the Noise: Click the Fetch/XHR filter. We don't care about CSS, images, or fonts. We only care about data.
Trigger the Request: Refresh the page. Watch the waterfall.

If nothing appears of note here, try different pages, try triggering pagination, loading and clicking buttons and watching to see what appears.

You are looking for requests that return JSON. They are often named intuitively, like graphql, search, products, or api. When you click "Preview" on these requests, you won't see HTML; you will see a structured object containing every piece of data you need—prices, descriptions, SKU numbers—already parsed and clean.

Pro Tip: Once you find a candidate URL, test it immediately in the browser console or URL bar. Try changing query parameters like page=1 to page=2. If the JSON response changes to show the next page of products, you have found your "Golden Endpoint."

Phase 2: The "Clean Room" Isolation

Finding the endpoint is only step one. Now you need to determine the minimum viable request required to access it programmatically.

Copy as cURL: Right-click the request in Chrome DevTools and select Copy as cURL.
Import to a Client: Open an API client like Bruno, Postman, or Insomnia. Import the cURL command.
The Baseline Test: Hit "Send." It should work perfectly because you are sending everything—every cookie, every header, and the exact session token your browser just generated.

The "Load-Bearing" Header Game

Efficient scrapers don't send 2KB of headers. You need to strip this down. Start unchecking headers one by one and resending the request:

Remove the Cookie header: Does it break? (Usually, yes).
Remove the Referer: Does it break? (Often, yes—sites check this to ensure the request came from their own frontend).
Remove the User-Agent: Does it break?
Check the Parameters: Can you change limit=10 to limit=100 to get more data in one shot?

Eventually, you will be left with the "skeleton key": the absolute minimum headers required to get a 200 OK. Usually, this consists of a User-Agent, a Referer, and a specific Auth Token or Session Cookie.

Phase 3: The Infrastructure Trap (The "Bonded" Token)

This is where most developers hit a wall. You take your cleaned-up request, put it into a Python script, and... 403 Forbidden.

Why? You have the right URL and the right headers.

In my analysis of modern scraping targets, I found that the API endpoint is increasingly performing a Cryptographic Binding check.

The IP Link: The Auth Token/Cookie you copied from your browser was generated for that specific IP address. When you run your script (likely on a server, VPN, or different proxy), the site sees a mismatch between the token's origin IP and your current request IP.
The Expiry Clock: These tokens are ephemeral. They are designed to expire, you will need to investigate how long.

If you are just looping through a list of URLs with a static token, you will burn out your access almost immediately.

Phase 4: Architecting the Solution

To make this work at scale, you cannot simply write a script. You need to build a Hybrid Architecture that manages state.

You need to engineer a system that takes the above into account and monitor its lifecycle:

The Storage Unit: You need a database (like Redis) to store a "Session Object." This object must contain:
- The Auth Token (Cookie).
- The IP Address used to generate it.
- The Creation Time.
The Browser Worker: You need a headless browser (Nodriver/Camoufox) to visit the site, execute the JavaScript, generate the token, and save it to your Storage Unit.
The HTTP Worker: Your actual scraper. It doesn't browse; it pulls the Token + IP combination from storage and hits the API directly.
The Rotation Logic: You need logic that checks the token age.
- Is the token older than 5 minutes? Stop.
- Spin up the Browser Worker.
- Generate a new Token.
- Update the Storage Unit.
- Resume scraping.

The Hidden Overhead

Suddenly, your simple scraping job requires a Proxy Management System (to ensure the Browser and HTTP worker share the same IP), a Browser Management System (to handle the heavy lifting of token generation), and a State Manager.

This is why "just scraping the API" is harder than it looks. The code to fetch the data is minimal—often just one function. But the infrastructure required to maintain the identity required to access that data is massive.

At Zyte, we abstract this entire architecture. Our API handles the browser fingerprinting, the IP, and the session rotation automatically. You simply send us the URL, and we handle the "Hybrid" complexity in the background, delivering you the clean JSON response without the infrastructure headache.

Want more? Join our community

The Modern Scrapy Developer's Guide (Part 3): Auto-Generating Page Objects with Web Scraping Co-pilot

John Rooney — Tue, 16 Dec 2025 18:55:04 +0000

Welcome to Part 3 of our Modern Scrapy series.

That refactor was a huge improvement, but it was still a lot of manual work. We had to:

Manually create our BookItem and BookListPage schemas.
Manually create the bookstoscrape_com.py Page Object file.
Manually use scrapy shell to find all the CSS selectors.
Manually write all the @field parsers.

What if you could do all of that in about 30 seconds?

In this guide, we'll show you how to use the Web Scraping Co-pilot (our VS Code extension) to automatically write 100% of your Items, Page Objects, and even your unit tests. We'll take our simple spider from Part 1 and upgrade it to the professional scrapy-poet architecture from Part 2, but this time, the AI will do all the heavy lifting.

Prerequisites & Setup

This tutorial assumes you have:

Completed Part 1 (see above)
Visual Studio Code installed.
The Web Scraping Co-pilot extension (which we'll install now).

Step 1: Installing Web Scraping Co-pilot

Inside VS Code, go to the "Extensions" tab and search for Web Scraping Co-pilot (published by Zyte).

Once installed, you'll see a new icon in your sidebar. Open it, and it will automatically detect your Scrapy project. It may ask to install a few dependencies like pytest—allow it to do so. This setup process ensures your environment is ready for AI-powered generation.

Step 2: Auto-Generating our `BookItem`

Let's start with the spider from Part 1. Our goal is to create a Page Object for our BookItem and add even more fields than we did in Part 2.

In the Co-pilot chat window:

Select "Web Scraping."
Write a prompt like this:

"Create a page object for the item BookItem using the sample URL https://books.toscrape.com/catalogue/the-host_979/index.html"

The co-pilot will now:

Check your project: It will confirm you have scrapy-poet and pytest (and will offer to install them if you don't).
Add scrapy-poet settings: It will automatically add the ADDONS and SCRAPY_POET_DISCOVER settings to your settings.py file.
Create your items.py: It will create a new BookItem class, but this time it will intelligently add all the fields it can find on the page.

# tutorial/items.py (Auto-Generated!)
import attrs

@attrs.define
class BookItem:
    """
    The structured data we extract from a book *detail* page.
    """
    name: str
    price: str
    url: str
    availability: str  # <-- New!
    number_of_reviews: int # <-- New!
    upc: str             # <-- New!

Create Fixtures: It creates a fixtures folder with the saved HTML and expected JSON output for testing.
Write the Page Object: It creates the tutorial/pages/bookstoscrape_com.py file and writes the entire Page Object, complete with all parsing logic and selectors, for all the new fields.

# tutorial/pages/bookstoscrape_com.py (Auto-Generated!)

from web_poet import WebPage, handle_urls, field, returns
from tutorial.items import BookItem

@handle_urls("[books.toscrape.com/catalogue](https://books.toscrape.com/catalogue)")
@returns(BookItem)
class BookDetailPage(WebPage):
    """
    This Page Object handles parsing data from book detail pages.
    """

    @field
    def name(self) -> str:
        return self.response.css("h1::text").get()

    @field
    def price(self) -> str:
        return self.response.css("p.price_color::text").get()

    @field
    def url(self) -> str:
        return self.response.url

    # All of this was written for us!
    @field
    def availability(self) -> str:
        return self.response.css("p.availability::text").getall()[1].strip()

    @field
    def number_of_reviews(self) -> int:
        return int(self.response.css("table tr:last-child td::text").get())

    @field
    def upc(self) -> str:
        return self.response.css("table tr:first-child td::text").get()

In 30 seconds, the Co-pilot has done everything we did manually in Part 2, but better—it even added more fields.

Step 3: Running the AI-Generated Tests

The best part? The Co-pilot also wrote unit tests for you. It created a tests folder with test_bookstoscrape_com.py.

You can just click "Run Tests" in the Co-pilot UI (or run pytest in your terminal).

$ pytest
================ test session starts ================
...
tests/test_bookstoscrape_com.py::test_book_detail[book_0] PASSED
tests/test_bookstoscrape_com.py::test_book_detail[book_1] PASSED
...
================ 8 tests passed in 0.10s ================

Your parsing logic is now fully tested, and you didn't write a single line of test code.

Step 4: Refactoring the Spider (The Easy Way)

Now, we just update our tutorial/spiders/books.py to use this new architecture, just like in Part 2.

# tutorial/spiders/books.py

import scrapy
# Import our new, auto-generated Item class
from tutorial.items import BookItem

class BooksSpider(scrapy.Spider):
    name = "books"
    # ... (rest of spider from Part 1) ...

    async def parse_listpage(self, response):
        product_urls = response.css("article.product_pod h3 a::attr(href)").getall()
        for url in product_urls:
            # We just tell Scrapy to call parse_book
            yield response.follow(url, callback=self.parse_book)

        next_page_url = response.css("li.next a::attr(href)").get()
        if next_page_url:
            yield response.follow(next_page_url, callback=self.parse_listpage)

    # We ask for the BookItem, and scrapy-poet does the rest!
    async def parse_book(self, response, book: BookItem):
        yield book

Step 5: Auto-Generating our `BookListPage`

We can repeat the exact same process for our list page to finish the refactor.

Prompt the Co-pilot:

"Create a page object for the list item BookListPage using the sample URL https://books.toscrape.com/catalogue/category/books/fantasy_19/index.html"

Result:

The Co-pilot will create the BookListPage item in items.py.
It will create the BookListPageObject in bookstoscrape_com.py with the parsers for book_urls and next_page_url.
It will write and pass the tests.

Now we can update our spider one last time to be fully architected.

# tutorial/spiders/books.py (FINAL VERSION)

import scrapy
from tutorial.items import BookItem, BookListPage # Import both

class BooksSpider(scrapy.Spider):
    # ... (name, allowed_domains, url) ...

    async def start(self):
        yield scrapy.Request(self.url, callback=self.parse_listpage)

    # We now ask for the BookListPage item!
    async def parse_listpage(self, response, page: BookListPage):

        # All parsing logic is GONE from the spider.
        for url in page.book_urls:
            yield response.follow(url, callback=self.parse_book)

        if page.next_page_url:
            yield response.follow(page.next_page_url, callback=self.parse_listpage)

    async def parse_book(self, response, book: BookItem):
        yield book

Our spider is now just a "crawler." It has zero parsing logic. All the hard work of finding selectors and writing parsers was automated by the Co-pilot.

Conclusion: The "Hybrid Developer"

The Web Scraping Co-pilot doesn't replace you. It accelerates you. It automates the 90% of work that is "grunt work" (finding selectors, writing boilerplate, creating tests) so you can focus on the 10% of work that matters: crawling logic, strategy, and handling complex sites.

This is how we, as the maintainers of Scrapy, build spiders professionally.

What's Next? Join the Community.

What's Next? Join the Community.
💬 TALK: Stuck on this Scrapy code? Ask the maintainers and 5k+ devs in our Discord.
▶️ WATCH: This post was based on our video! Watch the full walkthrough on our YouTube channel.
📩 READ: Want more? In Part 2, we'll cover Scrapy Items and Pipelines. Get the Extract newsletter so you don't miss it.

The Modern Scrapy Developer's Guide (Part 2): Page Objects with scrapy-poet

John Rooney — Tue, 16 Dec 2025 18:49:10 +0000

Welcome to Part 2 of our Modern Scrapy series. In Part 1, we built a working spider that crawls and scrapes an entire category. But if you look at our code, it's already getting messy. Our parse_listpage and parse_book functions are mixing two different jobs:

Crawling Logic: Finding the next page and following links.
Parsing Logic: Finding the data (name, price) with CSS selectors.

What happens when a selector changes? Or when you want to test your parsing logic? You have to run the whole spider. This is slow, hard to maintain, and difficult to test.

In this guide, we'll fix this by refactoring our spider to a professional, modern standard using Scrapy Items and Page Objects (via scrapy-poet). We will completely separate our crawling logic from our parsing logic. This will make our code cleaner, infinitely easier to test, and scalable.

What We'll Build

We will refactor our spider from Part 1. The spider itself will only handle crawling (following links). All the parsing logic will be moved into dedicated "Page Object" classes. scrapy-poet will automatically inject the correct, parsed item into our spider.

Look at how clean our spider's parse_book function becomes:

# The NEW parse_book function
# Where did the parsing logic go?! (Hint: scrapy-poet)

    async def parse_book(self, response, book: BookItem):
        # 'book' is a BookItem, magically injected and parsed
        # by scrapy-poet before this function is even called.
        # We just yield it.
        yield book

Prerequisites

This tutorial builds directly on Part 1: Building Your First Crawling Spider. Please complete that guide first, as we will be modifying the spider we built there.

Step 1: The "Why" (Separation of Concerns)

Our current spider is a monolith. The BooksSpider class knows how to crawl (find next page links, find product links) and how to parse (extract h1 tags, extract p.price_color).

This is bad. If we want to reuse our parsing logic, or test it without re-crawling the web, we can't.

The "Page Object" pattern solves this.

The Spider's Job: Crawling. Its only job is to navigate from page to page and yield Requests or Items.
The Page Object's Job: Parsing. Its only job is to take a response and extract structured data from it.

scrapy-poet is a library that automatically connects our spider to the correct Page Object.

Step 2: Create Our "Schema" (Scrapy Items)

First, let's define the data we're scraping. Instead of messy dictionaries, we'll use Scrapy Items. Scrapy comes with attrs, a fantastic library for this.

Open tutorial/items.py and add two classes: one for our book data and one for our list page data.

# tutorial/items.py

import attrs

@attrs.define
class BookItem:
    """
    The structured data we extract from a book *detail* page.
    """
    name: str
    price: str
    url: str

@attrs.define
class BookListPage:
    """
    The data and links we extract from a *list* page.
    """
    book_urls: list
    next_page_url: str | None

This is our "schema." It makes our code type-safe and easier to read.

Step 3: Install and Configure `scrapy-poet`

scrapy-poet is a separate package we need to install.

# Install scrapy-poet
uv add scrapy-poet
# or: pip install scrapy-poet

Now, we must enable it in tutorial/settings.py.

# tutorial/settings.py

# Add this to enable the scrapy-poet add-on
ADDONS = {
    'scrapy_poet.Addon': 300,
}

# Add this to tell scrapy-poet where to find our Page Objects
# 'tutorial.pages' means a folder named 'pages' in our 'tutorial' module
SCRAPY_POET_DISCOVER = ['tutorial.pages']

Step 4: Create Page Objects for Parsing

Now for the magic. Let's create the tutorial/pages module.

mkdir tutorial/pages
touch tutorial/pages/__init__.py

Inside this new folder, create a file named bookstoscrape_com.py. This file will hold all the parsing logic for bookstoscrape.com.

This is the most complex part, but it's a "set it and forget it" pattern.

# tutorial/pages/bookstoscrape_com.py

from web_poet import WebPage, handle_urls, field, returns

# Import our Item schemas
from tutorial.items import BookItem, BookListPage

# This class handles all book DETAIL pages
@handle_urls("[books.toscrape.com/catalogue](https://books.toscrape.com/catalogue)")
@returns(BookItem)
class BookDetailPage(WebPage):
    """
    This Page Object handles parsing data from book detail pages.
    """

    # The @field decorator tells scrapy-poet: "run this function
    # and put the result into the 'name' field of the BookItem."
    @field
    def name(self) -> str:
        # This is our parsing logic from Part 1
        return self.response.css("h1::text").get()

    @field
    def price(self) -> str:
        # This is our parsing logic from Part 1
        return self.response.css("p.price_color::text").get()

    @field
    def url(self) -> str:
        return self.response.url

# This class handles all book LIST pages (categories)
@handle_urls("[books.toscrape.com/catalogue/category](https://books.toscrape.com/catalogue/category)")
@returns(BookListPage)
class BookListPageObject(WebPage):
    """
    This Page Object handles parsing data from category/list pages.
    """

    @field
    def book_urls(self) -> list:
        # This is our parsing logic from Part 1
        return self.response.css("article.product_pod h3 a::attr(href)").getall()

    @field
    def next_page_url(self) -> str | None:
        # This is our parsing logic from Part 1
        return self.response.css("li.next a::attr(href)").get()

Look at that! All our messy response.css() calls are now neatly organized in their own classes, completely separate from our spider. The @handle_urls decorator tells scrapy-poet which Page Object to use for which URL.

Step 5: Refactor the Spider (The Payoff)

Now, let's go back to tutorial/spiders/books.py and refactor it. It becomes much simpler.

# tutorial/spiders/books.py

import scrapy
# Import our new Item classes
from tutorial.items import BookItem, BookListPage

class BooksSpider(scrapy.Spider):
    name = "books"
    allowed_domains = ["toscrape.com"]
    url: str = "https://books.toscrape.com/catalogue/category/books/fantasy_19/index.html"

    async def start(self):
        # We still start the same way
        yield scrapy.Request(self.url, callback=self.parse_listpage)

    # The 'page: BookListPage' is new.
    # We ask for the BookListPage item, and scrapy-poet injects it.
    async def parse_listpage(self, response, page: BookListPage):

        # 1. Get the parsed book URLs from the Page Object
        for url in page.book_urls:
            # We follow each URL, but our callback no longer
            # needs to do any work!
            yield response.follow(url, callback=self.parse_book)

        # 2. Get the next page URL from the Page Object
        if page.next_page_url:
            yield response.follow(page.next_page_url, callback=self.parse_listpage)

    # The 'book: BookItem' is new.
    # We ask for the BookItem, and scrapy-poet injects it.
    async def parse_book(self, response, book: BookItem):
        # Our parsing logic is GONE.
        # The 'book' variable is already a fully-populated
        # BookItem, parsed by our BookDetailPage Page Object.

        # We just yield it.
        yield book

Our spider is now only responsible for crawling. All parsing is handled by scrapy-poet and our Page Objects. This code is clean, testable, and incredibly easy to read.

When you run scrapy crawl books -o books.json, the output will be identical to Part 1, but your architecture is now 100x better.

The "Hard Part": Why This Still Breaks

We've built a professional, well-architected Scrapy spider. But we've just made a cleaner version of a spider that will still fail on a real-world site.

This architecture is beautiful, but it doesn't solve the "real" problems:

❌ IP Blocks: You're still hitting the site from one IP. You will be blocked.
❌ CAPTCHAs: You have no way to avoid captchas, and your spider will fail.
❌ JavaScript: If the prices were loaded by JS, our response.css() selectors would find nothing.

We've just organized our failing code.

The "Easy Way": Zyte API as a Universal Page Object
scrapy-poet is a great way to organise your scrapy code, making your projects easier to build, collaborate and maintain. However, it doesn't change the fact we are not doing anything to avoid web scraping bans.

So we can add the below settings using our Zyte API account to run our scrapy project through Zyte API.

# add scrapy-zyte-api python library
uv add scrapy-zyte-api
# settings.py
ZYTE_API_KEY = "YOUR_API_KEY"

ADDONS = {
    "scrapy_zyte_api.Addon": 500,
}

This is the power of combining a great architecture (Scrapy) with a powerful service (Zyte API).

Conclusion & Next Steps

Today you elevated your spider from a simple script to a professional-grade crawler. You learned the "Separation of Concerns" principle, defined data with Items, and separated parsing logic with scrapy-poet's Page Objects.

This is the modern way to build robust, testable, and scalable Scrapy spiders.

What's Next? Join the Community.
💬 TALK: Stuck on this Scrapy code? Ask the maintainers and 5k+ devs in our Discord.
▶️ WATCH: This post was based on our video! Watch the full walkthrough on our YouTube channel.
📩 READ: Want more? In Part 2, we'll cover Scrapy Items and Pipelines. Get the Extract newsletter so you don't miss it.

And if you're ready to skip the "Hard Part" entirely, get your free API key and try the "Easy Way."

Start Your Free Zyte Trial

The Modern Scrapy Developer's Guide (Part 1): Building Your First Spider

John Rooney — Tue, 16 Dec 2025 18:41:29 +0000

Scrapy can feel daunting. It's a massive, powerful framework, and the documentation can be overwhelming for a newcomer. Where do you even begin?

In this definitive guide, we will walk you through, step-by-step, how to build a real, multi-page crawling spider. You will go from an empty folder to a clean JSON file of structured data in about 15 minutes. We'll use modern, async/await Python and cover project setup, finding selectors, following links (crawling), and saving your data.

What We'll Build

We will build a Scrapy spider that crawls the "Fantasy" category on books.toscrape.com, follows the "Next" button to crawl every page in that category, follows the link for every book, and scrapes the name, price, and URL from all 48 books, saving the result to a clean books.json file.

Here's a preview of our final spider code:

# The final spider we'll build
import scrapy

class BooksSpider(scrapy.Spider):
    name = "books"
    allowed_domains = ["toscrape.com"]

    url: str = "https://books.toscrape.com/catalogue/category/books/fantasy_19/index.html](https://books.toscrape.com/catalogue/category/books/fantasy_19/index.html"

    async def start(self):
        yield scrapy.Request(self.url, callback=self.parse_listpage)

    async def parse_listpage(self, response):
        product_urls = response.css("article.product_pod h3 a::attr(href)").getall()
        for url in product_urls:
            yield response.follow(url, callback=self.parse_book)

        next_page_url = response.css("li.next a::attr(href)").get()
        if next_page_url:
            yield response.follow(next_page_url, callback=self.parse_listpage)

    async def parse_book(self, response):
        yield {
            "name": response.css("h1::text").get(),
            "price": response.css("p.price_color::text").get(),
            "url": response.url
        }

Prerequisites & Setup

Before we start, you'll need Python 3.x installed. We'll also be using a virtual environment to keep our dependencies clean. You can use standard pip or a modern package manager like uv.

First, let's create a project folder and activate a virtual environment.

# Create a new folder
mkdir scrapy_project
cd scrapy_project

# Option 1: Using standard pip + venv
python -m venv .venv
source .venv/bin/activate  # On Windows, use: .venv\Scripts\activate

# Option 2: Using uv (a fast, modern alternative)
uv init

Now, let's install Scrapy.

# Option 1: Using pip
pip install scrapy

# Option 2: Using uv
uv add scrapy
source .venv/bin/activate

Step 1: Initialize Your Project

With Scrapy installed, we can use its built-in command-line tools to generate our project boilerplate.

First, create the project itself.

# The 'scrapy startproject' command creates the project structure
# The '.' tells it to use the current folder
scrapy startproject tutorial .

You'll see a tutorial folder and a scrapy.cfg file appear. This folder contains all your project's logic.

Next, we'll generate our first spider.

# The 'genspider' command creates a new spider file
# Usage: scrapy genspider <spider_name> <allowed_domain>
scrapy genspider books toscrape.com

If you look in tutorial/spiders/, you'll now see books.py. This is where we'll write our code.

Step 2: Configure Your Settings

Before we write our spider, let's quickly adjust two settings in tutorial/settings.py.

ROBOTSTXT_OBEY

By default, Scrapy respects robots.txt files. This is a good practice, but our test site (toscrape.com) doesn't have one, which can cause a 404 error in our logs. We'll turn it off for this tutorial.

# tutorial/settings.py

# Find this line and change it to False
ROBOTSTXT_OBEY = False

Concurrency

Scrapy is polite by default and runs slowly. Since toscrape.com is a test site built for scraping, we can speed it up.

# tutorial/settings.py

# Uncomment or add these lines
CONCURRENT_REQUESTS = 16
DOWNLOAD_DELAY = 0

Warning: These settings are for this test site only. When scraping in the wild, you must be mindful of your target site and use respectful DOWNLOAD_DELAY and CONCURRENT_REQUESTS values.

Step 3: Finding Our Selectors (with `scrapy shell`)

To scrape a site, we need to tell Scrapy what data to get. We do this with CSS selectors. The scrapy shell is the best tool for this.

Let's launch the shell on our target category page:

scrapy shell https://books.toscrape.com/catalogue/category/books/fantasy_19/index.html

This will download the page and give you an interactive shell with a response object.

You can even type view(response) to open the page in your browser exactly as Scrapy sees it!

Let's find the data we need:

Find all Book Links:

By inspecting the page, we see each book is in an article.product_pod. The link is inside an h3.

# In scrapy shell:
>>> response.css("article.product_pod h3 a::attr(href)").getall()
[
  '../../../../the-host_979/index.html',
  '../../../../the-hunted_978/index.html',
  ...
]

That getall() gives us a clean list of all the URLs.

Find the "Next" Page Link:

At the bottom, we find the "Next" button in an li.next.

# In scrapy shell:
>>> response.css("li.next a::attr(href)").get()
'page-2.html'

This get() gives us the single link we need for pagination.

Find the Book Data (on a product page):

Finally, let's open a shell on a product page to find the selectors for our data.

# Exit the shell and open a new one:
scrapy shell https://books.toscrape.com/catalogue/the-host_979/index.html

# In scrapy shell:
>>> response.css("h1::text").get()
'The Host'

>>> response.css("p.price_color::text").get()
'£25.82'

Perfect. We now have all the selectors we need.

Step 4: Building the Spider (Crawling & Parsing)

Now, let's open tutorial/spiders/books.py and write our spider. We'll use the user's provided code, as it's a clean, final version.

Delete the boilerplate in books.py and replace it with this:

# tutorial/spiders/books.py

import scrapy

class BooksSpider(scrapy.Spider):
    name = "books"
    allowed_domains = ["toscrape.com"]

    # This is our starting URL (the first page of the Fantasy category)
    url: str = "https://books.toscrape.com/catalogue/category/books/fantasy_19/index.html"

    # This is the modern, async version of 'start_requests'
    # It's called once when the spider starts.
    async def start(self):
        # We yield our first request, sending the response to 'parse_listpage'
        yield scrapy.Request(self.url, callback=self.parse_listpage)

    # This function handles the *category page*
    async def parse_listpage(self, response):

        # 1. Get all product URLs using the selector we found
        product_urls = response.css("article.product_pod h3 a::attr(href)").getall()

        # 2. For each product URL, follow it and send the response to 'parse_book'
        for url in product_urls:
            yield response.follow(url, callback=self.parse_book)

        # 3. Find the 'Next' page URL
        next_page_url = response.css("li.next a::attr(href)").get()

        # 4. If a 'Next' page exists, follow it and send the response
        #    back to *this same function*
        if next_page_url:
            yield response.follow(next_page_url, callback=self.parse_listpage)

    # This function handles the *product page*
    async def parse_book(self, response):

        # We yield a dictionary of the data we want
        yield {
            "name": response.css("h1::text").get(),
            "price": response.css("p.price_color::text").get(),
            "url": response.url
        }

This code is clean and efficient. response.follow is smart enough to handle the relative URLs (like page-2.html) for us.

Step 5: Running The Spider & Saving Data

We're ready to run. Go to your terminal (at the project root) and run:

scrapy crawl books

You'll see Scrapy start up, and in the logs, you'll see all 48 items being scraped!

But we want to save this data. Scrapy has a built-in "Feed Exporter" that makes this easy. We just use the -o (output) flag.

scrapy crawl books -o books.json

This will run the spider again, but this time, you'll see a new books.json file in your project root, containing all 48 items, perfectly structured.

Conclusion & Next Steps

Today you built a powerful, modern, async Scrapy crawler. You learned how to set up a project, find selectors, follow links, and handle pagination.

This is just the starting block.

What's Next? Join the Community.

💬 TALK: Stuck on this Scrapy code? Ask the maintainers and 5k+ devs in our Discord.

▶️ WATCH: This post was based on our video! Watch the full walkthrough on our YouTube channel.

📩 READ: Want more? In Part 2, we'll cover Scrapy Items and Pipelines. Get the Extract newsletter so you don't miss it.

DEV Community: Zyte

Stop Scraping HTML - There's a better way.

Phase 1: The Discovery (XHR Filtering)

Phase 2: The "Clean Room" Isolation

The "Load-Bearing" Header Game

Phase 3: The Infrastructure Trap (The "Bonded" Token)

Phase 4: Architecting the Solution

The Hidden Overhead

The Modern Scrapy Developer's Guide (Part 3): Auto-Generating Page Objects with Web Scraping Co-pilot

Prerequisites & Setup

Step 1: Installing Web Scraping Co-pilot

Step 2: Auto-Generating our BookItem

Step 3: Running the AI-Generated Tests

Step 4: Refactoring the Spider (The Easy Way)

Step 5: Auto-Generating our BookListPage

Conclusion: The "Hybrid Developer"

The Modern Scrapy Developer's Guide (Part 2): Page Objects with scrapy-poet

What We'll Build

Prerequisites

Step 1: The "Why" (Separation of Concerns)

Step 2: Create Our "Schema" (Scrapy Items)

Step 3: Install and Configure scrapy-poet

Step 4: Create Page Objects for Parsing

Step 5: Refactor the Spider (The Payoff)

The "Hard Part": Why This Still Breaks

Conclusion & Next Steps

The Modern Scrapy Developer's Guide (Part 1): Building Your First Spider

What We'll Build

Prerequisites & Setup

Step 1: Initialize Your Project

Step 2: Configure Your Settings

Step 3: Finding Our Selectors (with scrapy shell)

Step 4: Building the Spider (Crawling & Parsing)

Step 5: Running The Spider & Saving Data

Conclusion & Next Steps

What's Next? Join the Community.

Step 2: Auto-Generating our `BookItem`

Step 5: Auto-Generating our `BookListPage`

Step 3: Install and Configure `scrapy-poet`

Step 3: Finding Our Selectors (with `scrapy shell`)