Handling Pagination in Scrapy: Scrape Every Page Without Breaking

The first time I tried scraping a paginated site, I only got the first page. I knew there were 50 pages, but my spider stopped after page 1.

I didn't know about response.follow() or pagination patterns. I was manually building URLs and getting it wrong.

Once I learned the pagination patterns, scraping multi-page sites became trivial. Let me show you every pagination pattern and how to handle it.

---

Why Pagination Matters

Most websites split content across pages:

• Product listings (page 1, 2, 3...)
• Search results
• Blog archives
• Category pages

If you don't handle pagination:

• Miss most of the data
• Only scrape first page
• Incomplete results

---

Pattern 1: Next Button (Most Common)

Website has a "Next" button linking to the next page.

HTML Example

<a class="next" href="/products?page=2">Next</a>

Spider Code

import scrapy

class NextButtonSpider(scrapy.Spider):
    name = 'next_button'
    start_urls = ['https://example.com/products']

    def parse(self, response):
        # Scrape items on current page
        for product in response.css('.product'):
            yield {
                'name': product.css('h2::text').get(),
                'price': product.css('.price::text').get()
            }

        # Follow next page link
        next_page = response.css('.next::attr(href)').get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

Key points:

• response.follow() handles relative URLs automatically
• Check if next_page exists before following
• Use same callback (self.parse) to repeat on next page

What the Docs Don't Tell You

Different "Next" selectors:

# Try multiple selectors
next_page = response.css('.next::attr(href)').get()
next_page = next_page or response.css('a.pagination-next::attr(href)').get()
next_page = next_page or response.css('a[rel="next"]::attr(href)').get()
next_page = next_page or response.xpath('//a[contains(text(), "Next")]/@href').get()

---

Pattern 2: Page Numbers (1, 2, 3...)

Website shows page numbers with links.

HTML Example

<div class="pagination">
    <a href="/products?page=1">1</a>
    <a href="/products?page=2">2</a>
    <a href="/products?page=3">3</a>
</div>

Spider Code

class PageNumberSpider(scrapy.Spider):
    name = 'page_numbers'
    start_urls = ['https://example.com/products?page=1']

    def parse(self, response):
        # Scrape items
        for product in response.css('.product'):
            yield {
                'name': product.css('h2::text').get()
            }

        # Follow all pagination links
        for page_link in response.css('.pagination a::attr(href)').getall():
            yield response.follow(page_link, callback=self.parse)

Scrapy automatically deduplicates URLs, so this won't visit the same page twice!

---

Pattern 3: Known Number of Pages

You know there are exactly N pages.

Spider Code

class KnownPagesSpider(scrapy.Spider):
    name = 'known_pages'

    def start_requests(self):
        base_url = 'https://example.com/products?page={}'

        # Scrape pages 1 through 50
        for page_num in range(1, 51):
            url = base_url.format(page_num)
            yield scrapy.Request(url, callback=self.parse)

    def parse(self, response):
        for product in response.css('.product'):
            yield {
                'name': product.css('h2::text').get()
            }

Simple and fast!

---

Pattern 4: Infinite Scroll (AJAX Loading)

Page loads more content as you scroll down.

Method 1: Find the AJAX API

Most infinite scroll sites load data via AJAX. Find the API:

1. Open DevTools (F12)
2. Network tab → XHR filter
3. Scroll down
4. Look for JSON responses

Example API found:

https://example.com/api/products?offset=0&limit=20
https://example.com/api/products?offset=20&limit=20

Spider:

import json

class InfiniteScrollSpider(scrapy.Spider):
    name = 'infinite_scroll'

    def start_requests(self):
        url = 'https://example.com/api/products?offset=0&limit=20'
        yield scrapy.Request(url, callback=self.parse_api)

    def parse_api(self, response):
        data = json.loads(response.text)

        # Extract items
        for product in data['products']:
            yield {
                'name': product['name'],
                'price': product['price']
            }

        # Check if more data
        if data['has_more']:
            # Get next offset
            offset = int(response.url.split('offset=')[1].split('&')[0])
            next_offset = offset + 20

            next_url = f'https://example.com/api/products?offset={next_offset}&limit=20'
            yield scrapy.Request(next_url, callback=self.parse_api)

Method 2: Use Playwright to Scroll

If no API is available, use Playwright to scroll:

import scrapy

class ScrollSpider(scrapy.Spider):
    name = 'scroll'

    def start_requests(self):
        yield scrapy.Request(
            'https://example.com/products',
            meta={
                'playwright': True,
                'playwright_include_page': True
            },
            callback=self.parse
        )

    async def parse(self, response):
        page = response.meta['playwright_page']

        # Scroll down 10 times
        for i in range(10):
            await page.evaluate('window.scrollTo(0, document.body.scrollHeight)')
            await page.wait_for_timeout(2000)  # Wait 2 seconds

        # Get final HTML
        content = await page.content()
        await page.close()

        # Parse with Scrapy
        from scrapy.http import HtmlResponse
        new_response = HtmlResponse(
            url=response.url,
            body=content.encode('utf-8')
        )

        for product in new_response.css('.product'):
            yield {
                'name': product.css('h2::text').get()
            }

---

Pattern 5: "Load More" Button

Button that loads more items via AJAX.

Find the API

Same as infinite scroll: check Network tab when clicking "Load More".

Example:

POST https://example.com/load-more
payload: {"page": 2}

Spider:

class LoadMoreSpider(scrapy.Spider):
    name = 'load_more'

    def start_requests(self):
        yield scrapy.Request(
            'https://example.com/products',
            callback=self.parse
        )

    def parse(self, response):
        # Scrape initial items
        for product in response.css('.product'):
            yield {
                'name': product.css('h2::text').get()
            }

        # Simulate "Load More" clicks
        for page_num in range(2, 11):  # Load 10 more times
            yield scrapy.FormRequest(
                'https://example.com/load-more',
                formdata={'page': str(page_num)},
                callback=self.parse_more
            )

    def parse_more(self, response):
        data = json.loads(response.text)

        for product in data['products']:
            yield {
                'name': product['name']
            }

---

Pattern 6: Cursor-Based Pagination

API uses cursor tokens instead of page numbers.

Example Response

{
  "products": [...],
  "next_cursor": "abc123xyz"
}

Spider:

class CursorSpider(scrapy.Spider):
    name = 'cursor'

    def start_requests(self):
        url = 'https://api.example.com/products'
        yield scrapy.Request(url, callback=self.parse)

    def parse(self, response):
        data = json.loads(response.text)

        # Extract items
        for product in data['products']:
            yield {
                'name': product['name']
            }

        # Follow next cursor
        next_cursor = data.get('next_cursor')
        if next_cursor:
            next_url = f'https://api.example.com/products?cursor={next_cursor}'
            yield scrapy.Request(next_url, callback=self.parse)

---

Pattern 7: URL Parameters (Offset/Limit)

URLs use offset and limit parameters.

Example URLs

/products?offset=0&limit=20
/products?offset=20&limit=20
/products?offset=40&limit=20

Spider:

class OffsetSpider(scrapy.Spider):
    name = 'offset'

    def start_requests(self):
        # Start with offset 0
        url = 'https://example.com/products?offset=0&limit=20'
        yield scrapy.Request(url, callback=self.parse)

    def parse(self, response):
        products = response.css('.product')

        # Scrape items
        for product in products:
            yield {
                'name': product.css('h2::text').get()
            }

        # If we got items, there might be more
        if products:
            # Extract current offset
            offset = int(response.url.split('offset=')[1].split('&')[0])

            # Next offset
            next_offset = offset + 20
            next_url = f'https://example.com/products?offset={next_offset}&limit=20'

            yield scrapy.Request(next_url, callback=self.parse)

---

Pattern 8: Date-Based Pagination

Archive pages organized by date.

Example URLs

/archive/2024/01
/archive/2024/02
/archive/2024/03

Spider:

from datetime import datetime, timedelta

class DateSpider(scrapy.Spider):
    name = 'date'

    def start_requests(self):
        # Start date
        start_date = datetime(2024, 1, 1)
        end_date = datetime(2024, 12, 31)

        current_date = start_date

        while current_date <= end_date:
            url = f'https://example.com/archive/{current_date.year}/{current_date.month:02d}'
            yield scrapy.Request(url, callback=self.parse)

            # Next month
            current_date = current_date + timedelta(days=32)
            current_date = current_date.replace(day=1)

    def parse(self, response):
        for article in response.css('.article'):
            yield {
                'title': article.css('h2::text').get()
            }

---

Stopping Pagination

Don't scrape forever! Add stop conditions.

Stop After N Pages

class LimitedSpider(scrapy.Spider):
    name = 'limited'
    max_pages = 10
    page_count = 0

    def parse(self, response):
        self.page_count += 1

        # Scrape items
        for product in response.css('.product'):
            yield {'name': product.css('h2::text').get()}

        # Stop if reached limit
        if self.page_count >= self.max_pages:
            self.logger.info(f'Reached max pages: {self.max_pages}')
            return

        # Continue pagination
        next_page = response.css('.next::attr(href)').get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

Stop on Empty Page

def parse(self, response):
    products = response.css('.product')

    # If no products, stop
    if not products:
        self.logger.info('No more products, stopping')
        return

    # Scrape items
    for product in products:
        yield {'name': product.css('h2::text').get()}

    # Continue
    next_page = response.css('.next::attr(href)').get()
    if next_page:
        yield response.follow(next_page, callback=self.parse)

---

Common Mistakes

Mistake #1: Not Using response.follow()

# BAD (doesn't handle relative URLs)
next_page = response.css('.next::attr(href)').get()
yield scrapy.Request(next_page)  # Might be relative!

# GOOD
next_page = response.css('.next::attr(href)').get()
if next_page:
    yield response.follow(next_page)  # Handles relative URLs

Mistake #2: Creating Duplicate URLs

# BAD (might visit same page twice)
for page in range(1, 100):
    url = f'/page/{page}'
    yield scrapy.Request(url, dont_filter=True)  # Forces duplicates!

# GOOD
for page in range(1, 100):
    url = f'/page/{page}'
    yield scrapy.Request(url)  # Scrapy deduplicates automatically

Mistake #3: Not Checking If Next Page Exists

# BAD (crashes if no next page)
next_page = response.css('.next::attr(href)').get()
yield response.follow(next_page)  # next_page might be None!

# GOOD
next_page = response.css('.next::attr(href)').get()
if next_page:
    yield response.follow(next_page)

---

Testing Pagination

Make sure pagination works:

def test_follows_pagination():
    html = '''
    <div class="product">Product 1</div>
    <a class="next" href="/page2">Next</a>
    '''

    response = HtmlResponse(url='http://example.com', body=html.encode())

    results = list(spider.parse(response))

    # Should have item and next request
    items = [r for r in results if not isinstance(r, Request)]
    requests = [r for r in results if isinstance(r, Request)]

    assert len(items) == 1
    assert len(requests) == 1
    assert 'page2' in requests[0].url

---

Complete Real-World Example

Production-ready pagination handler:

import scrapy

class ProductionPaginationSpider(scrapy.Spider):
    name = 'pagination'
    start_urls = ['https://example.com/products']

    # Configuration
    max_pages = 100
    page_count = 0
    min_items_per_page = 5

    def parse(self, response):
        self.page_count += 1
        self.logger.info(f'Scraping page {self.page_count}')

        # Extract products
        products = response.css('.product')

        # Log if few items (might indicate end)
        if len(products) < self.min_items_per_page:
            self.logger.warning(
                f'Only {len(products)} items on page {self.page_count} '
                f'(expected at least {self.min_items_per_page})'
            )

        # If no products, we've reached the end
        if not products:
            self.logger.info('No products found, stopping pagination')
            return

        # Scrape all products
        for product in products:
            name = product.css('h2::text').get()
            price = product.css('.price::text').get()

            if name and price:
                yield {
                    'name': name.strip(),
                    'price': price.strip(),
                    'page': self.page_count,
                    'url': response.url
                }

        # Check if reached max pages
        if self.page_count >= self.max_pages:
            self.logger.info(f'Reached max pages limit: {self.max_pages}')
            return

        # Try multiple next page selectors
        next_page = (
            response.css('.next::attr(href)').get() or
            response.css('a.pagination-next::attr(href)').get() or
            response.css('a[rel="next"]::attr(href)').get() or
            response.xpath('//a[contains(text(), "Next")]/@href').get()
        )

        if next_page:
            self.logger.info(f'Following next page: {next_page}')
            yield response.follow(next_page, callback=self.parse)
        else:
            self.logger.info('No next page link found, stopping')

    def closed(self, reason):
        self.logger.info('='*60)
        self.logger.info('PAGINATION STATISTICS')
        self.logger.info(f'Total pages scraped: {self.page_count}')
        self.logger.info(f'Close reason: {reason}')
        self.logger.info('='*60)

---

Summary

Common pagination patterns:

1. Next button - Most common, use response.follow()
2. Page numbers - Follow all pagination links
3. Known pages - Generate URLs in start_requests()
4. Infinite scroll - Find AJAX API or use Playwright
5. Load more - POST request to load-more endpoint
6. Cursor-based - Follow next_cursor in API
7. Offset/limit - Increment offset parameter
8. Date-based - Generate date-based URLs

Best practices:

• Always use response.follow() for relative URLs
• Check if next page exists before following
• Add stop conditions (max pages or empty results)
• Log pagination progress
• Test pagination logic

Debugging tips:

• Check if next_page selector is correct
• Verify URLs are being generated correctly
• Watch for infinite loops
• Check Scrapy stats for request count

Remember:

• Scrapy deduplicates URLs automatically
• response.follow() handles relative URLs
• Stop when no more items found
• Log progress for debugging

Start with "Next" button pattern, it covers 80% of cases!

Happy scraping! 🕷️