Polite Scraper — Books to Scrape

#github #javascript #node #webscraping

A fully documented, ethical web scraper built for the FlyRank Internship project.

It demonstrates polite scraping practices, caching, validation, and reproducibility.

Target Classification

Site: Books to Scrape (https://books.toscrape.com)
Purpose: Public sandbox for scraping practice
Scope: First 3 catalogue pages (60 books total)
robots.txt: Accessible and allows crawling.
Ethical Statement: This scraper is used only for learning. I will not reuse this code on other sites without checking their rules and terms first.

Installation & Run

git clone https://github.com/ameh0429/scraper.git
cd scraper
npm install
node src/index.js

Runs in under 5 minutes and produces:

data/raw-books.json
output/books.json
output/errors.json
output/run-report.json

Record Schema

Each book record follows this structure:

Field	Type	Example
`title`	string	"A Light in the Attic"
`product_url`	string (URL)	`https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html`
`price_text`	string	"£51.77"
`price_gbp`	number	51.77
`availability_text`	string	"In stock (22 available)"
`rating_text`	string \	null
`description`	string \	null
`source_page`	string (URL)	`https://books.toscrape.com/catalogue/page-1.html`
`fetched_at`	string (ISO timestamp)	`"2026-08-18T16:25:17.386Z"`

Politeness Rules

Custom User‑Agent: FlyRankInternship-A9/1.0 (+https://github.com/ameh0429/scraper.git)
Timeout: 5 seconds per request
Delay: 500 ms between catalogue pages
Caching: HTML saved locally to avoid repeated requests
Retries: One retry for 5xx errors; skips 404/403 pages
Idempotency: Re‑running the scraper never duplicates data

Run evidence

{
  "start_time": "2026-08-18T16:25:17.386Z",
  "pages_fetched": 60,
  "cache_hits": 60,
  "valid_records": 60,
  "invalid_records": 0,
  "failed_pages": 0,
  "duration_seconds": 11.684
}

This report proves the scraper completed successfully and politely.
No browser automation was needed — the data is already in the HTML responses.

Project Stages

Classify Scraping Target — verify site and ethics
Fetch Once, Cache Once — polite single‑fetch caching
Find All Three Pages — dynamic catalogue discovery
Extract Raw Records — structured data collection
Survive Failures, Report Run — resilient error handling
Publish Evidence — reproducible, documented results

DEV Community