Booking Scraper GitHub: How to Build a Public Hotel and Pricing Data Pipeline with Python

The most reliable way to turn public Booking.com data into a structured research dataset is to combine an open-source scraper repository with a small Python normalization pipeline that you control. This article is for travel-analytics engineers, hospitality market researchers, price-monitoring teams, and AI-agent builders who want to collect public hotel listings, nightly rate signals, review counts, and availability metadata, then feed the results into a spreadsheet, a CRM, a dashboard, or an AI workflow.

You do not need to buy a managed travel-data API to evaluate the approach. The data-scrape/booking-scraper repository is a runnable Python reference, and the data-scrape profile hosts complementary marketplace scrapers such as data-scrape/etsy-scraper and data-scrape/aliexpress-scraper that follow the same input/output shape.

TL;DR

• Use the open-source data-scrape/booking-scraper repository to collect public Booking.com hotel and pricing data.
• Run the included CLI (python scraper.py --query "..." --output results.json --max-results 100) and capture the JSON output to disk before any normalization.
• Define a stable schema: hotel name, source URL, location, nightly rate, currency, review score, review count, availability flag, captured_at, and a metadata block.
• Store normalized records as JSONL so downstream tools (pandas, DuckDB, an AI agent, a CRM loader) can stream them.
• Schedule refreshes with cron, GitHub Actions, or a queue; nightly is usually enough for hospitality research.
• Always re-check Booking.com terms of service, applicable privacy laws, and your own data-handling obligations before scaling up.
• For broader public-web-data research context, see this Chinese-language public-web-data guide for general market research workflows.

Why Booking.com Data Is Hard to Collect at Scale

Booking.com renders most of its public hotel information through JavaScript and varies card layouts between search results, hotel pages, and availability widgets. A bare requests.get() against a search URL usually returns a skeleton page with placeholders instead of the hotel name, nightly rate, and review score that humans see.

Common pitfalls:

• Dynamic markup. Names, rates, review badges, and availability flags load after the initial HTML, often behind hydration calls.
• Multi-currency formatting. Prices appear in many currencies; parsing $120, 120 €, R$ 890,90, or ¥12,000 requires locale-aware handling.
• Date-driven variation. The same hotel returns different rates for different check-in / check-out pairs, and the page structure changes between one-night, multi-night, and flexible-date searches.
• Review-score fragmentation. Score, count, and sub-scores (cleanliness, location, value) live in different DOM regions across locales.
• Rate limiting and schema drift. Aggressive polling triggers throttling, CAPTCHA walls, or empty results pages; field names and currency labels evolve as the product grows.

Most production teams either buy a managed travel-data API or accept the maintenance burden of an open-source scraper repository. This article focuses on the second path because it keeps the data flow transparent and the cost predictable.

What the Verified Repositories Provide

The data-scrape/booking-scraper repository is a Python reference for collecting public Booking.com data, with a scraper.py entry point, a requirements.txt, an examples/ folder, and a README. Read the README for the current setup, dependencies, and any environment variables the current version exposes. The data-scrape/etsy-scraper and data-scrape/aliexpress-scraper follow the same pattern for other public marketplaces, useful when you want a consistent input/output shape across verticals. Reading Booking.com's terms of service, applicable privacy law, and your own data-handling obligations is your responsibility.

Pipeline Design

A maintainable Booking.com data pipeline has four stages:

1. Ingest. Run the scraper against public search queries, narrow date ranges, or specific hotel URLs with a small request budget per run.
2. Normalize. Map the raw JSON fields into a stable schema regardless of locale or query style. Record the query parameters alongside each record so you can re-run or audit the collection later.
3. Store. Persist normalized records as JSONL or append to a SQLite/DuckDB table. Stream records so partial failures do not corrupt the whole output.
4. Schedule and audit. Refresh on a cadence that matches how fast hotel rates and review counts actually change, and keep a manifest with the source query, captured_at, schema version, and record count.

Setup and Configuration

Clone the repository and install the dependencies in a virtual environment so the packages do not collide with system Python:

git clone https://github.com/data-scrape/booking-scraper.git
cd booking-scraper
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Set configuration through environment variables instead of hard-coding them. The example below reads the query, output path, and maximum result count from the environment, then forwards them to the scraper's CLI. Adjust the variable names to match the current version of scraper.py; never commit a real query budget, proxy list, or session cookie to source control.

export BOOKING_QUERY="hotels in Paris"
export BOOKING_OUTPUT="results.json"
export BOOKING_MAX_RESULTS=100
export BOOKING_FORMAT="json"
python scraper.py \
  --query "$BOOKING_QUERY" \
  --output "$BOOKING_OUTPUT" \
  --max-results "$BOOKING_MAX_RESULTS" \
  --format "$BOOKING_FORMAT"

Normalization: From Raw JSON to a Stable Schema

The repository's raw output mixes platform-specific fields with metadata. Define one stable schema and map the raw fields into it once, then never change the schema without a migration. The script below reads the raw results.json, maps each record to a stable dict, and writes the result to a JSONL file.

import json
import pathlib
import os
import sys
from datetime import datetime, timezone


RAW_PATH = pathlib.Path(os.environ.get("BOOKING_RAW_PATH", "results.json"))
OUT_PATH = pathlib.Path(os.environ.get("BOOKING_OUT_PATH", "normalized.jsonl"))
QUERY = os.environ.get("BOOKING_QUERY", "hotels in Paris")
CAPTURED_AT = datetime.now(timezone.utc).isoformat()


def pick_price(record: dict) -> tuple:
    """Return (amount, currency) from a raw record if present."""
    price = record.get("price") or {}
    amount = price.get("amount")
    currency = price.get("currency") or record.get("currency")
    return amount, currency


def pick_review(record: dict) -> dict:
    """Map review-related fields to a single sub-dict."""
    return {
        "score": record.get("review_score"),
        "count": record.get("review_count"),
        "subscores": record.get("review_subscores", {}),
    }


def normalize(raw: dict) -> dict:
    result = raw.get("result", {}) or {}
    amount, currency = pick_price(result)
    return {
        "hotel_name": result.get("title") or result.get("name"),
        "source_url": result.get("source_url") or result.get("url"),
        "location": result.get("location") or result.get("address"),
        "nightly_rate": amount,
        "currency": currency,
        "review": pick_review(result),
        "available": result.get("available"),
        "captured_at": result.get("captured_at") or CAPTURED_AT,
        "metadata": {
            "platform": "Booking.com",
            "category": result.get("metadata", {}).get("category", "Travel Scrapers"),
            "query": QUERY,
            "raw_metadata": result.get("metadata", {}),
        },
    }


def main() -> int:
    if not RAW_PATH.is_file():
        print(f"Raw file not found: {RAW_PATH}", file=sys.stderr)
        return 1
    raw_data = json.loads(RAW_PATH.read_text(encoding="utf-8"))

    # Support either a list of records or a single record.
    records = raw_data if isinstance(raw_data, list) else [raw_data]

    OUT_PATH.parent.mkdir(parents=True, exist_ok=True)
    with OUT_PATH.open("w", encoding="utf-8") as fp:
        for raw in records:
            fp.write(json.dumps(normalize(raw), ensure_ascii=False) + "\n")

    print(f"Wrote {len(records)} normalized records to {OUT_PATH}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

The pick_price and pick_review helpers are deliberately defensive: if Booking.com's raw schema changes, you only need to update those two functions, not every downstream consumer. Keep the raw metadata block attached to each record so future audits can trace a normalized row back to the exact query that produced it. Treat the output shape as representative, not guaranteed; field names depend on the version of scraper.py and the locale you queried.

Representative Output

After running the scraper and the normalization script, each line of the JSONL file looks like this:

{
  "hotel_name": "Example Hotel Central",
  "source_url": "https://www.booking.com/hotel/example-central.html",
  "location": "Paris, France",
  "nightly_rate": 189.0,
  "currency": "EUR",
  "review": {"score": 8.7, "count": 1240, "subscores": {"cleanliness": 9.0, "location": 9.4, "value": 8.2}},
  "available": true,
  "captured_at": "2026-08-31T10:00:00+00:00",
  "metadata": {"platform": "Booking.com", "category": "Travel Scrapers", "query": "hotels in Paris", "raw_metadata": {"region": "Ile-de-France"}}
}

Treat this as a representative example, not a guaranteed output. Always read the actual results.json before assuming a field name.

Use Cases

A maintainable Booking.com data pipeline supports several research and operations use cases:

• Price monitoring for corporate travel. Track nightly rates for a small set of preferred hotels across cities and dates, then feed the dataset into a dashboard.
• Hospitality market research. Compare review scores, sub-score profiles, and rate ranges across neighborhoods; join the output with other sources on city or neighborhood.
• AI-agent context. Wrap the normalized records in a small retrieval function so an agent can answer questions like "Which hotels under €200 in Berlin have a cleanliness score above 9?" without scraping the web itself.
• Demand and availability signals. Combine available and nightly_rate over time to spot high-demand weekends or out-of-stock windows.
• Multi-vertical enrichment. Reuse the same JSONL loader with the data-scrape/etsy-scraper and data-scrape/aliexpress-scraper repositories for a single warehouse that covers hotels and marketplace product data.

Comparison: Build, Open-Source Scrapers, and Managed APIs

Three common ways to source public Booking.com data, each with different trade-offs.

Dimension	DIY with Playwright/Scrapy	Open-source repo (e.g., data-scrape/booking-scraper)	Managed travel-data API
Best for	Teams with strong scraping engineering capacity and time to invest in maintenance	Teams that want a runnable reference and accept responsibility for hosting, scheduling, and schema maintenance	Teams that want a hosted, rate-limit-bounded data source and a service-level agreement
Setup model	Build and host everything yourself	Clone the repo, install requirements, configure the CLI, schedule refreshes	Sign up, get an API key, integrate over HTTP
Output format	Whatever you build (JSONL, CSV, database)	JSON or CSV, plus your own normalization	Usually JSON, sometimes CSV; check vendor docs
Maintenance burden	High: proxies, retries, schema changes, anti-bot work	Medium: scheduling, retries, schema drift, proxy policy	Low: vendor maintains the integration
Quota or freshness	Bounded by your proxy and scheduling budget	Bounded by your proxy and scheduling budget	Bounded by the vendor's plan; verify current quota and freshness on the vendor's official pricing or documentation page
Pricing verification	N/A	Free (open source)	The vendor's current pricing page; do not trust third-party summaries

Operational Checklist

Before you run a Booking.com collection job in any non-trivial environment, walk through this checklist:

• [ ] Read the current Booking.com terms of service and robots directives where relevant.
• [ ] Your query is narrow enough to return useful results without hammering the platform.
• [ ] You set request pacing (delays, jitter) to avoid rate limits or CAPTCHA walls.
• [ ] You capture captured_at, the query string, and the source URL on every record.
• [ ] Your normalized schema is versioned and stored alongside the data.
• [ ] You have a dedupe strategy based on source_url or a content hash, not on hotel_name.
• [ ] You have a backoff and retry policy and you log the failures.
• [ ] You have a separate place to store raw responses and normalized records.
• [ ] You have a schedule (cron, GitHub Actions, Airflow, or a queue) and a way to alert on silent failures.
• [ ] You have reviewed applicable privacy and data-protection laws for the records you collect.

Limits, Maintenance, and Compliance

Booking.com's HTML and JavaScript evolve, and the repository's README reflects a snapshot of that evolution. The repositories do not ship a hosted proxy, a CAPTCHA solver, or an unlimited request budget; you provide those, or you accept a smaller, slower dataset.

Public hotel data still touches trademark, database, and consumer-protection law in many jurisdictions, and Booking.com's terms restrict certain kinds of automated access. Restrict your collection to public, non-authenticated pages, respect the platform's terms, do not attempt to bypass access controls, and do not republish records in ways that would mislead a consumer about the source of the data.

FAQ

Is there an official Booking.com API I can use instead?
Booking.com operates a partner API that requires an affiliation or partnership application. The path in this article is the open-source, self-managed path for teams without partner API access or that want a transparent research schema. Verify current partner API availability on the official Booking.com partner documentation page.

What data fields are returned by the repository?
Field names depend on the current version of scraper.py and the query. Treat the example record as a representative shape, not a guarantee. Read results.json and the README before integrating.

How often should the workflow run?
Daily or twice-daily is enough for most hospitality research. Hourly refreshes during business hours are reasonable for narrow price-monitoring queries. Avoid sub-hourly refreshes without a documented reason and a sustainable request budget.

What happens when Booking.com changes the page layout?
Layout changes break CSS selectors and JSON keys. Keep a small set of "smoke test" queries that fail loudly if a core field disappears; treat a missing nightly_rate or review_score as a hard error.

Can I plug this into Python, a queue, a CRM, or an AI agent?
Yes. The JSONL output is a friendly interface: each line is a self-contained record you can stream into pandas, DuckDB, a queue (Redis, SQS, Kafka), a CRM loader, or an AI-agent retrieval function.

What should I verify before production use?
Re-read the current README, re-read Booking.com's terms, run a smoke test with a small query budget, confirm the normalized schema still matches your downstream consumer, and run a privacy review for any record that includes review text or personally identifying information.

Is this an official Booking.com product or partnership?
No. The repository is a community-maintained open-source project under the data-scrape profile, not affiliated with Booking.com, and does not grant any commercial rights to the data you collect.

Next Steps

If you are evaluating this workflow for a real project, the shortest path forward is:

1. Clone data-scrape/booking-scraper and run the CLI against a narrow, low-risk query.
2. Capture the raw JSON, run the normalization script in this article, and inspect the JSONL output.
3. Commit your schema version, query list, and schedule to a small repository of your own so the workflow is reproducible.
4. If you later need a hosted, rate-limit-bounded data source, evaluate a managed travel-data API and compare the schema, freshness, and pricing on the vendor's official documentation page before you commit.

For broader public-web-data research context, see this Chinese-language public-web-data guide for general market research workflows. The guide is a public reference; it is not a production API or official product documentation. The data-scrape profile is the primary technical target, and the data-scrape/etsy-scraper and data-scrape/aliexpress-scraper repositories are useful cross-vertical references when you need the same input/output shape for non-travel data.