Etsy Scraper GitHub: How to Build a Product and Shop Data Pipeline with Python

The most reliable way to turn public Etsy marketplace data into a structured research dataset is to combine an open-source scraper repository with a small Python normalization pipeline that you control. This article is for marketplace analysts, e-commerce developers, growth researchers, and data engineers who want to collect public product listings, shop profiles, and review counts, normalize the fields into a stable schema, and store the results in a format that downstream tools can ingest.

You do not need to pay per-result for a managed marketplace API or maintain your own headless-browser farm from scratch. The data-scrape/etsy-scraper repository provides a runnable reference implementation, and the broader data-scrape profile hosts complementary e-commerce scraper repositories. Your job is to wrap the scraper in a pipeline that deduplicates records, enforces a schema, and respects platform terms.

TL;DR

• Use the open-source data-scrape/etsy-scraper repository to collect public Etsy listings and shop profile JSON.
• Normalize the raw JSON into a consistent schema: listing URL, listing ID, shop handle, title, price, currency, tags, and timestamp.
• Store normalized records as JSONL so downstream tools can stream them without parsing a large array.
• Schedule the workflow with cron, a scheduler, or a queue; refresh cadence depends on how fast listings and reviews change.
• Always verify current rate limits, robots directives, and Etsy's terms of service before running at scale.
• For broader public-web-data research context, see this Chinese-language public-web-data guide for market research.

Why Etsy Data Is Hard to Collect at Scale

Etsy's public pages render most listing metadata through JavaScript, mix structured markup with dynamic fragments, and shift card layouts when the platform experiments with new shopfront designs. A bare requests.get() call against a listing URL usually returns a skeleton page with placeholders instead of the price, shop, and tag data that humans see in the browser.

The common pitfalls are:

• Dynamic markup: Listing titles, shop handles, prices, and review counts load after the initial HTML.
• Currency and locale formatting: Etsy shows prices in many currencies. Parsing $12.50 vs 12,50 € vs £9.99 requires locale-aware handling, not naive string splitting.
• Review and favorite counts: Aggregated counts are loaded asynchronously and may differ from per-listing snapshots.
• Rate limiting: Aggressive polling from one IP triggers throttling, CAPTCHA walls, or empty pages.
• Schema drift: Field names, attribute layouts, and tag taxonomies evolve as the platform grows.
• Seller-vs-shop terminology: A shop (the storefront) and a listing (an individual item) live in different schema branches, and mapping the two together is easy to get wrong.

Because of this, most production teams either buy a managed marketplace data API or use an open-source scraper repository and accept the maintenance burden. This article focuses on the second path because it keeps the data flow transparent and the schema under your control.

What the Verified Repositories Provide

The data-scrape/etsy-scraper repository is an open-source reference for collecting public Etsy listings through a scraper-style wrapper. It is useful when you want a structured request/response pattern and plan to integrate the scraper into a larger research or analytics workflow. Read the repository's README for the current setup instructions, dependencies, and any environment variables it requires.

For comparison, the data-scrape/aliexpress-scraper and data-scrape/booking-scraper repositories follow a similar pattern for other public marketplaces. They are useful reference points when you want to compare normalization logic across vertical-specific shops.

None of these repositories are unlimited services. Do not assume that a repository example shows a live, rate-limit-free endpoint. Plan for retries, backoff, and a quota of requests that depends on your proxy or session strategy.

Pipeline Design

A maintainable Etsy data pipeline has four stages:

1. Ingest. Run the scraper repository against a list of public listing URLs, shop handles, or category search pages.
2. Normalize. Map the raw JSON fields into a stable schema regardless of which repository produced them.
3. Deduplicate. Drop listings you already captured in the previous run using the listing ID.
4. Store and expose. Write JSONL files or load the records into a database, queue, or analytics tool.

This design isolates schema drift to the normalization layer. When Etsy changes a field name or currency format, you update one mapper instead of rewriting every downstream query.

Environment Setup

Create a project directory and install the dependencies the repository lists. Most require Python 3.10 or newer plus requests, httpx, or a headless-browser driver. For the normalization layer you only need the standard library and a JSON processor.

python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Define environment variables for anything that changes between environments:

export ETSY_INPUT_LIST="etsy_listings.txt"
export ETSY_OUTPUT_DIR="./data"
export ETSY_RUN_ID="2026-08-23"
export ETSY_USER_AGENT="etsy-pipeline/1.0 ([e-mail])"

Keep proxy credentials, cookies, and any required session tokens out of the code. If the repository requires a session cookie or signed request helper, load it from the environment and rotate it on the schedule recommended by the documentation.

Runnable Python Normalization Workflow

The script below does not call a live Etsy endpoint. It reads representative JSON records that a scraper repository would produce and turns them into a clean, deduplicated JSONL file. Replace INPUT_PATH with the actual output directory of your chosen repository.

import json
import os
import re
from datetime import datetime, timezone
from decimal import Decimal, InvalidOperation
from pathlib import Path

INPUT_PATH = os.environ.get("ETSY_INPUT_LIST", "sample_etsy_records.json")
OUTPUT_DIR = Path(os.environ.get("ETSY_OUTPUT_DIR", "./data"))
RUN_ID = os.environ.get("ETSY_RUN_ID", datetime.now(timezone.utc).strftime("%Y-%m-%d"))

OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
OUTPUT_FILE = OUTPUT_DIR / f"etsy_normalized_{RUN_ID}.jsonl"
SEEN_FILE = OUTPUT_DIR / "seen_listing_ids.txt"

LISTING_SCHEMA = {
    "listing_url": None,
    "listing_id": None,
    "title": None,
    "shop_handle": None,
    "shop_name": None,
    "price_amount": None,
    "price_currency": None,
    "tags": [],
    "review_count": None,
    "favorite_count": None,
    "published_at": None,
    "scraped_at": None,
    "source_repo": None,
}


def parse_price(raw_price, raw_currency):
    """Return (Decimal, ISO currency) so the JSON output stays serializable."""
    if raw_price is None:
        return None, None
    cleaned = re.sub(r"[^0-9.,\-]", "", str(raw_price))
    cleaned = cleaned.replace(",", ".")
    try:
        amount = Decimal(cleaned)
    except InvalidOperation:
        return None, raw_currency
    return amount, raw_currency


def normalize_listing(raw, source_repo):
    """Map a raw repository record to the stable schema."""
    record = LISTING_SCHEMA.copy()

    record["listing_url"] = (
        raw.get("url") or raw.get("listing_url") or raw.get("share_url")
    )
    record["listing_id"] = (
        str(raw.get("id") or raw.get("listing_id") or raw.get("listingId"))
    )
    record["title"] = raw.get("title") or raw.get("name") or raw.get("listing_title")

    # Shop may be nested or flat depending on the scraper version.
    if isinstance(raw.get("shop"), dict):
        record["shop_handle"] = raw["shop"].get("handle") or raw["shop"].get("username")
        record["shop_name"] = raw["shop"].get("name") or raw["shop"].get("display_name")
    else:
        record["shop_handle"] = raw.get("shop_handle") or raw.get("seller")
        record["shop_name"] = raw.get("shop_name")

    amount, currency = parse_price(
        raw.get("price") or raw.get("price_amount"),
        raw.get("currency") or raw.get("price_currency"),
    )
    record["price_amount"] = str(amount) if amount is not None else None
    record["price_currency"] = currency

    tags = raw.get("tags")
    if isinstance(tags, str):
        tags = [t.strip() for t in tags.split(",") if t.strip()]
    record["tags"] = tags or []

    record["review_count"] = raw.get("review_count") or raw.get("reviews")
    record["favorite_count"] = raw.get("favorite_count") or raw.get("favorites")
    record["published_at"] = raw.get("published_at") or raw.get("creation_timestamp")
    record["scraped_at"] = datetime.now(timezone.utc).isoformat()
    record["source_repo"] = source_repo
    return record


def load_seen_ids(path):
    if not path.exists():
        return set()
    return set(path.read_text(encoding="utf-8").splitlines())


def save_seen_ids(path, ids):
    path.write_text("\n".join(sorted(ids)), encoding="utf-8")


# ---------------------------------------------------------------------------
# Example: load raw records produced by a scraper repository.
# In production, point this at the actual repository output file or queue.
# ---------------------------------------------------------------------------
sample_records = [
    {
        "id": "1122334455",
        "url": "https://www.etsy.com/listing/1122334455/handmade-ceramic-mug",
        "title": "Handmade Ceramic Mug, 12 oz",
        "shop": {"handle": "demoMaker", "name": "Demo Maker Studio"},
        "price": "$24.50",
        "currency": "USD",
        "tags": ["ceramic", "kitchen", "handmade"],
        "review_count": 138,
        "favorite_count": 412,
        "creation_timestamp": "2026-07-04T08:15:00+00:00",
    },
    {
        "id": "1122334456",
        "url": "https://www.etsy.com/listing/1122334456/linen-apron-natural",
        "title": "Linen Apron, Natural Color",
        "shop_handle": "kitchentools",
        "shop_name": "Kitchen Tools Co.",
        "price": "€18,00",
        "currency": "EUR",
        "tags": "linen,apron,kitchen",
        "reviews": 22,
        "favorites": 60,
        "creation_timestamp": "2026-08-01T11:42:00+00:00",
    },
]

seen_ids = load_seen_ids(SEEN_FILE)
new_records = []

for raw in sample_records:
    record = normalize_listing(raw, source_repo="etsy-scraper")
    lid = record.get("listing_id")
    if lid and lid in seen_ids:
        continue
    if lid:
        seen_ids.add(lid)
    new_records.append(record)

with open(OUTPUT_FILE, "w", encoding="utf-8") as f:
    for record in new_records:
        # Convert Decimal to float for JSON compliance.
        serializable = {
            k: (float(v) if k == "price_amount" and v is not None else v)
            for k, v in record.items()
        }
        f.write(json.dumps(serializable, ensure_ascii=False) + "\n")

save_seen_ids(SEEN_FILE, seen_ids)

print(f"Wrote {len(new_records)} new records to {OUTPUT_FILE}")
print(f"Total unique listing IDs tracked: {len(seen_ids)}")

The mapper uses fallback chains for every field. If a repository renames shop.handle to seller_handle, the script still captures the value. The price parser handles both $24.50 and €18,00 because Etsy's currency output mixes formats by region.

Representative Output

After normalization, each line in etsy_normalized_2026-08-23.jsonl looks like this:

{
  "listing_url": "https://www.etsy.com/listing/1122334455/handmade-ceramic-mug",
  "listing_id": "1122334455",
  "title": "Handmade Ceramic Mug, 12 oz",
  "shop_handle": "demoMaker",
  "shop_name": "Demo Maker Studio",
  "price_amount": 24.50,
  "price_currency": "USD",
  "tags": ["ceramic", "kitchen", "handmade"],
  "review_count": 138,
  "favorite_count": 412,
  "published_at": "2026-07-04T08:15:00+00:00",
  "scraped_at": "2026-08-23T10:15:00+00:00",
  "source_repo": "etsy-scraper"
}

JSONL is a good default because you can append new records without rewriting the whole file, stream it into pandas or DuckDB, and load it into a queue such as RabbitMQ or SQS with minimal parsing overhead.

Build vs. Buy Checklist

Dimension	Open-source scraper repository	Managed marketplace data API
Best for	Teams that need full schema control and can maintain the scraper	Teams that need data immediately without infrastructure work
Setup	Clone repo, install deps, configure proxies/sessions	Sign up, copy endpoint, set API key
Data coverage	Depends on the repository and current site structure	Depends on provider coverage; confirm before buying
Output format	Raw or custom normalized JSON/JSONL	Usually structured JSON with fixed schema
Maintenance burden	High: updates needed when site markup changes	Low to medium: provider handles breakage
Integration path	Local scripts, cron, queues, custom API wrapper	Direct HTTP API, SDK if available
Rate/freshness	Set by your proxy and session strategy	Set by provider plan; verify current limits
Pricing	Infrastructure cost only; proxies are usually paid	Per-request or per-result; confirm on official pricing page

There is no universally better choice. A solo analyst tracking one niche of handmade goods may be fine with an open-source repository. A marketplace intelligence firm tracking thousands of shops per day will usually prefer a managed API once the cost of maintenance exceeds the subscription price.

Business Use Cases

Public Etsy listing and shop data supports several workflows:

• Trend detection. Track tag velocity and category growth across a cohort of listings to spot emerging handmade niches.
• Competitive research. Compare pricing, review velocity, and favorite counts across peer shops in the same category.
• Catalog enrichment. Append Etsy shop metadata to an internal product catalog for cross-marketplace analysis.
• Content strategy. Identify what tags and title patterns correlate with high review counts in a given category.
• Influencer and seller vetting. Compare public shop signals before approaching sellers for collaborations or wholesale.

In every case, stay within the boundaries of public data. Do not attempt to access private shops, direct messages, or data that requires authentication beyond what the shop owner has made public.

Compliance and Maintenance Notes

Etsy's terms of service, robots directives, and regional privacy laws apply regardless of whether you use an open-source tool or a paid API. Before running any scraper at scale:

• Read the current Etsy Terms of Service and robots.txt behavior.
• Collect only public listings and public shop profile data. Private shops, member-only data, and non-public seller information are out of scope.
• Respect rate limits. Add backoff, jitter, and retry logic. Do not hammer the platform from a single IP.
• Keep a log of what you collected, when, and from which public URL. This audit trail matters for compliance and debugging.
• Rotate session credentials and proxies on the schedule recommended by the repository documentation.
• Monitor schema drift. Schedule a small weekly job that alerts you when expected fields disappear or change type.
• Honor robots.txt directives for etsy.com. Etsy is explicit in its policies about automated access, so verify what is permitted before scaling up.

A scraper is never maintenance-free. Plan for at least a few hours per month of upkeep, more if Etsy makes significant changes to its listing or shop layout.

FAQ

Is there an official Etsy API for this data?
Etsy offers an open API for sellers and selected partners, but third-party research use cases typically require approval and fit within documented rate limits. The repository-based workflow described here is for public web data that does not rely on official API access.

What data fields are returned?
The exact fields depend on the repository and the current page structure. Common fields include listing URL, listing ID, title, shop handle, shop display name, price, currency, tags, review count, favorite count, and creation timestamp. Always inspect a sample response before building downstream logic.

Can I use this for private shops or direct messages?
No. This workflow is for public listings and public shop profiles only. Private shops, non-public listings, and member-only data are out of scope and should not be attempted.

How often should the workflow run?
For trend tracking, every 4 to 24 hours is usually enough. For active monitoring of a small set of competitor shops, hourly may be justified. Match the cadence to the listing velocity and your rate-limit headroom.

What happens when Etsy changes its page layout?
The scraper repository may stop returning the expected fields. Your normalization layer will log missing fields, giving you a clear signal of what broke. Update the mapper or switch to a newer repository version when available.

Can this connect to n8n, a CRM, or an AI agent?
Yes. Once the data is in JSONL or loaded into a database, you can feed it into n8n workflows, CRM enrichment tools, or LLM context windows. The data-scrape profile hosts related repositories for business-data and AI-agent workflows.

What should I verify before production use?
Confirm that the repository you chose is currently maintained, that your proxy and session strategy complies with Etsy's policies, that your output schema matches downstream requirements, and that you have monitoring and alerting in place for failures and schema drift.

What's Next

If you want a transparent, maintainable way to collect public Etsy data, start with the verified repositories:

• data-scrape/etsy-scraper for public Etsy listing and shop metadata extraction.
• data-scrape/aliexpress-scraper for adjacent marketplace coverage.
• data-scrape for related open-source scraping tools.

For additional public-web-data research context, see the Chinese-language public-web-data guide for market research, which covers cross-marketplace research patterns in more depth.

Build the normalization layer first, run it on a small sample, and only scale once the schema and compliance checks are solid.