AliExpress Scraper GitHub: How to Build a Product and Seller Data Pipeline with Python

The most reliable way to turn public AliExpress marketplace data into a structured research dataset is to combine an open-source scraper repository with a small Python normalization pipeline that you control. This article is for marketplace analysts, dropshipping researchers, e-commerce developers, and competitive-intelligence teams who want to collect public product listings, seller signals, and price metadata, normalize the fields into a stable schema, and store the results in a format that downstream tools can ingest.

You do not need to pay per-result for a managed marketplace API or maintain your own headless-browser farm from scratch. The data-scrape/aliexpress-scraper repository provides a runnable reference implementation, and the broader data-scrape profile hosts complementary e-commerce and travel scraper repositories. Your job is to wrap the scraper in a pipeline that deduplicates records, enforces a schema, and respects platform terms.

TL;DR

• Use the open-source data-scrape/aliexpress-scraper repository to collect public AliExpress listing and seller JSON.
• Normalize the raw JSON into a consistent schema: product URL, product ID, title, price, currency, seller name, rating, shipping tags, and timestamp.
• Store normalized records as JSONL so downstream tools can stream them without parsing a large array.
• Schedule the workflow with cron, a scheduler, or a queue; refresh cadence depends on how fast listings and seller ratings change.
• Always verify current rate limits, robots directives, and AliExpress terms of service before running at scale.
• For broader public-web-data research context, see this Chinese-language public-web-data guide for e-commerce price monitoring.

Why AliExpress Data Is Hard to Collect at Scale

AliExpress public pages render much of their product metadata through JavaScript, mix structured markup with dynamic fragments, and vary card layouts between search results, category pages, and product detail pages. A bare requests.get() call against a search URL often returns a skeleton page with placeholders instead of the price, shipping estimate, and seller rating that humans see in the browser.

The common pitfalls are:

• Dynamic markup: Product titles, prices, discount badges, and shipping tags load after the initial HTML.
• Multi-currency and locale formatting: AliExpress shows prices in many currencies and formats. Parsing $12.50 vs 12,50 € vs R$ 89,90 requires locale-aware handling, not naive string splitting.
• Seller metadata fragmentation: Seller name, store rating, and positive-feedback percentage may appear in different page regions depending on whether you are on a search result or detail page.
• Shipping and variant data: Shipping options, color/size variants, and bundle discounts are often loaded asynchronously and may differ by destination country.
• Rate limiting: Aggressive polling from one IP triggers throttling, CAPTCHA walls, or empty pages.
• Schema drift: Field names, attribute layouts, and category taxonomies evolve as the platform grows.

Because of this, most production teams either buy a managed marketplace data API or use an open-source scraper repository and accept the maintenance burden. This article focuses on the second path because it keeps the data flow transparent and the schema under your control.

What the Verified Repositories Provide

The data-scrape/aliexpress-scraper repository is an open-source reference for collecting public AliExpress listings through a scraper-style wrapper. It is useful when you want a structured request/response pattern and plan to integrate the scraper into a larger research or analytics workflow. Read the repository's README for the current setup instructions, dependencies, and any environment variables it requires.

For comparison, the data-scrape/booking-scraper and data-scrape/etsy-scraper repositories follow a similar pattern for other public marketplaces. They are useful reference points when you want to compare normalization logic across vertical-specific shops.

None of these repositories are unlimited services. Do not assume that a repository example shows a live, rate-limit-free endpoint. Plan for retries, backoff, and a quota of requests that depends on your proxy or session strategy.

Pipeline Design

A maintainable AliExpress data pipeline has four stages:

1. Ingest. Run the scraper repository against a list of public product URLs, search result pages, or category pages.
2. Normalize. Map the raw JSON fields into a stable schema regardless of which repository produced them.
3. Deduplicate. Drop products you already captured in the previous run using the product ID.
4. Store and expose. Write JSONL files or load the records into a database, queue, or analytics tool.

This design isolates schema drift to the normalization layer. When AliExpress changes a field name or currency format, you update one mapper instead of rewriting every downstream query.

Environment Setup

Create a project directory and install the dependencies the repository lists. Most require Python 3.10 or newer plus requests, httpx, or a headless-browser driver. For the normalization layer you only need the standard library and a JSON processor.

python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Define environment variables for anything that changes between environments:

export ALIEXPRESS_INPUT_LIST="aliexpress_listings.txt"
export ALIEXPRESS_OUTPUT_DIR="./data"
export ALIEXPRESS_RUN_ID="2026-08-28"
export ALIEXPRESS_USER_AGENT="aliexpress-pipeline/1.0 ([e-mail])"

Keep proxy credentials, cookies, and any required session tokens out of the code. If the repository requires a session cookie or signed request helper, load it from the environment and rotate it on the schedule recommended by the documentation.

Runnable Python Normalization Workflow

The script below does not call a live AliExpress endpoint. It reads representative JSON records that a scraper repository would produce and turns them into a clean, deduplicated JSONL file. Replace INPUT_PATH with the actual output directory of your chosen repository.

import json
import os
import re
from datetime import datetime, timezone
from decimal import Decimal, InvalidOperation
from pathlib import Path

INPUT_PATH = os.environ.get("ALIEXPRESS_INPUT_LIST", "sample_aliexpress_records.json")
OUTPUT_DIR = Path(os.environ.get("ALIEXPRESS_OUTPUT_DIR", "./data"))
RUN_ID = os.environ.get("ALIEXPRESS_RUN_ID", datetime.now(timezone.utc).strftime("%Y-%m-%d"))

OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
OUTPUT_FILE = OUTPUT_DIR / f"aliexpress_normalized_{RUN_ID}.jsonl"
SEEN_FILE = OUTPUT_DIR / "seen_product_ids.txt"

PRODUCT_SCHEMA = {
    "product_url": None,
    "product_id": None,
    "title": None,
    "seller_name": None,
    "seller_rating": None,
    "seller_positive_feedback": None,
    "price_amount": None,
    "price_currency": None,
    "original_price": None,
    "discount_rate": None,
    "shipping_tags": [],
    "review_count": None,
    "order_count": None,
    "published_at": None,
    "scraped_at": None,
    "source_repo": None,
}


def parse_price(raw_price, raw_currency):
    """Return (Decimal, ISO currency) so the JSON output stays serializable."""
    if raw_price is None:
        return None, None
    cleaned = re.sub(r"[^0-9.,\-]", "", str(raw_price))
    # Normalize European comma-decimal to dot-decimal for Decimal parsing.
    if "," in cleaned and "." in cleaned:
        cleaned = cleaned.replace(",", "")
    elif "," in cleaned:
        cleaned = cleaned.replace(",", ".")
    try:
        amount = Decimal(cleaned)
    except InvalidOperation:
        return None, raw_currency
    return amount, raw_currency


def normalize_product(raw, source_repo):
    """Map a raw repository record to the stable schema."""
    record = PRODUCT_SCHEMA.copy()

    record["product_url"] = (
        raw.get("url") or raw.get("product_url") or raw.get("share_url")
    )
    record["product_id"] = (
        str(raw.get("id") or raw.get("product_id") or raw.get("itemId"))
    )
    record["title"] = raw.get("title") or raw.get("name") or raw.get("product_title")

    # Seller may be nested or flat depending on the scraper version.
    if isinstance(raw.get("seller"), dict):
        record["seller_name"] = raw["seller"].get("name") or raw["seller"].get("store_name")
        record["seller_rating"] = raw["seller"].get("rating")
        record["seller_positive_feedback"] = raw["seller"].get("positive_feedback")
    else:
        record["seller_name"] = raw.get("seller_name") or raw.get("store_name")
        record["seller_rating"] = raw.get("seller_rating")
        record["seller_positive_feedback"] = raw.get("seller_positive_feedback")

    amount, currency = parse_price(
        raw.get("price") or raw.get("sale_price") or raw.get("price_amount"),
        raw.get("currency") or raw.get("price_currency"),
    )
    record["price_amount"] = str(amount) if amount is not None else None
    record["price_currency"] = currency

    original, _ = parse_price(raw.get("original_price"), currency)
    record["original_price"] = str(original) if original is not None else None

    discount = raw.get("discount_rate") or raw.get("discount")
    if discount is not None:
        record["discount_rate"] = str(discount)

    shipping = raw.get("shipping_tags") or raw.get("shipping")
    if isinstance(shipping, str):
        shipping = [s.strip() for s in shipping.split(",") if s.strip()]
    record["shipping_tags"] = shipping or []

    record["review_count"] = raw.get("review_count") or raw.get("reviews")
    record["order_count"] = raw.get("order_count") or raw.get("orders")
    record["published_at"] = raw.get("published_at") or raw.get("creation_timestamp")
    record["scraped_at"] = datetime.now(timezone.utc).isoformat()
    record["source_repo"] = source_repo
    return record


def load_seen_ids(path):
    if not path.exists():
        return set()
    return set(path.read_text(encoding="utf-8").splitlines())


def save_seen_ids(path, ids):
    path.write_text("\n".join(sorted(ids)), encoding="utf-8")


# ---------------------------------------------------------------------------
# Example: load raw records produced by a scraper repository.
# In production, point this at the actual repository output file or queue.
# ---------------------------------------------------------------------------
sample_records = [
    {
        "id": "1005001234567890",
        "url": "https://www.aliexpress.com/item/1005001234567890.html",
        "title": "Wireless Bluetooth Earbuds, Noise Cancelling, 40H Playtime",
        "seller": {"name": "GadgetStore", "rating": "4.7", "positive_feedback": "95.2%"},
        "price": "$18.99",
        "currency": "USD",
        "original_price": "$32.99",
        "discount_rate": "42%",
        "shipping_tags": ["Free shipping", "Estimated delivery Aug 30"],
        "review_count": 2847,
        "order_count": 15320,
        "creation_timestamp": "2025-11-12T09:30:00+00:00",
    },
    {
        "id": "1005000987654321",
        "url": "https://www.aliexpress.com/item/1005000987654321.html",
        "title": "Mechanical Keyboard Keycaps, PBT, 104 Keys",
        "seller_name": "KeyCapHub",
        "seller_rating": "4.9",
        "seller_positive_feedback": "98.1%",
        "price": "€24,50",
        "currency": "EUR",
        "shipping_tags": "Standard shipping, Estimated delivery Sep 05",
        "review_count": 412,
        "order_count": 1890,
        "creation_timestamp": "2026-02-28T14:15:00+00:00",
    },
]

seen_ids = load_seen_ids(SEEN_FILE)
new_records = []

for raw in sample_records:
    record = normalize_product(raw, source_repo="aliexpress-scraper")
    pid = record.get("product_id")
    if pid and pid in seen_ids:
        continue
    if pid:
        seen_ids.add(pid)
    new_records.append(record)

with open(OUTPUT_FILE, "w", encoding="utf-8") as f:
    for record in new_records:
        # Convert Decimal strings to numeric values for JSON compliance.
        serializable = {}
        for k, v in record.items():
            if k in {"price_amount", "original_price"} and v is not None:
                serializable[k] = float(v)
            else:
                serializable[k] = v
        f.write(json.dumps(serializable, ensure_ascii=False) + "\n")

save_seen_ids(SEEN_FILE, seen_ids)

print(f"Wrote {len(new_records)} new records to {OUTPUT_FILE}")
print(f"Total unique product IDs tracked: {len(seen_ids)}")

The mapper uses fallback chains for every field. If a repository renames seller.name to store_name, the script still captures the value. The price parser handles both $18.99 and €24,50 because AliExpress currency output mixes formats by region.

Representative Output

After normalization, each line in aliexpress_normalized_2026-08-28.jsonl looks like this:

{
  "product_url": "https://www.aliexpress.com/item/1005001234567890.html",
  "product_id": "1005001234567890",
  "title": "Wireless Bluetooth Earbuds, Noise Cancelling, 40H Playtime",
  "seller_name": "GadgetStore",
  "seller_rating": "4.7",
  "seller_positive_feedback": "95.2%",
  "price_amount": 18.99,
  "price_currency": "USD",
  "original_price": 32.99,
  "discount_rate": "42%",
  "shipping_tags": ["Free shipping", "Estimated delivery Aug 30"],
  "review_count": 2847,
  "order_count": 15320,
  "published_at": "2025-11-12T09:30:00+00:00",
  "scraped_at": "2026-08-28T10:00:00+00:00",
  "source_repo": "aliexpress-scraper"
}

JSONL is a good default because you can append new records without rewriting the whole file, stream it into pandas or DuckDB, and load it into a queue such as RabbitMQ or SQS with minimal parsing overhead.

Build vs. Buy Checklist

Dimension	Open-source scraper repository	Managed marketplace data API
Best for	Teams that need full schema control and can maintain the scraper	Teams that need data immediately without infrastructure work
Setup	Clone repo, install deps, configure proxies/sessions	Sign up, copy endpoint, set API key
Data coverage	Depends on the repository and current site structure	Depends on provider coverage; confirm before buying
Output format	Raw or custom normalized JSON/JSONL	Usually structured JSON with fixed schema
Maintenance burden	High: updates needed when site markup changes	Low to medium: provider handles breakage
Integration path	Local scripts, cron, queues, custom API wrapper	Direct HTTP API, SDK if available
Rate/freshness	Set by your proxy and session strategy	Set by provider plan; verify current limits
Pricing	Infrastructure cost only; proxies are usually paid	Per-request or per-result; confirm on official pricing page

There is no universally better choice. A solo dropshipping researcher tracking one niche may be fine with an open-source repository. A marketplace intelligence firm tracking thousands of SKUs per day will usually prefer a managed API once the cost of maintenance exceeds the subscription price.

Business Use Cases

Public AliExpress listing and seller data supports several workflows:

• Price monitoring. Track price drops, discount campaigns, and currency-adjusted pricing across a cohort of products.
• Supplier research. Compare seller ratings, positive-feedback percentages, and order counts to shortlist reliable suppliers.
• Competitive intelligence. Monitor which titles, shipping tags, and discount rates competitors use in the same category.
• Dropshipping validation. Identify products with high order counts and stable seller ratings before adding them to a store.
• Trend detection. Track category velocity and emerging keywords to spot products that are gaining traction.

In every case, stay within the boundaries of public data. Do not attempt to access private seller dashboards, order history, or data that requires authentication beyond what the listing page makes public.

Compliance and Maintenance Notes

AliExpress terms of service, robots directives, and regional privacy laws apply regardless of whether you use an open-source tool or a paid API. Before running any scraper at scale:

• Read the current AliExpress Terms of Use and robots.txt behavior.
• Collect only public listings and public seller profile data. Private dashboards, member-only data, and non-public seller information are out of scope.
• Respect rate limits. Add backoff, jitter, and retry logic. Do not hammer the platform from a single IP.
• Keep a log of what you collected, when, and from which public URL. This audit trail matters for compliance and debugging.
• Rotate session credentials and proxies on the schedule recommended by the repository documentation.
• Monitor schema drift. Schedule a small weekly job that alerts you when expected fields disappear or change type.
• Honor robots.txt directives for aliexpress.com. Verify what is permitted before scaling up.

A scraper is never maintenance-free. Plan for at least a few hours per month of upkeep, more if AliExpress makes significant changes to its listing or seller layout.

FAQ

Is there an official AliExpress API for this data?
AliExpress offers partner APIs for sellers and selected developers, but third-party research use cases typically require approval and fit within documented rate limits. The repository-based workflow described here is for public web data that does not rely on official API access.

What data fields are returned?
The exact fields depend on the repository and the current page structure. Common fields include product URL, product ID, title, seller name, seller rating, positive feedback percentage, price, original price, discount rate, shipping tags, review count, order count, and creation timestamp. Always inspect a sample response before building downstream logic.

Can I use this for private seller dashboards or order data?
No. This workflow is for public listings and public seller signals only. Private dashboards, order history, and member-only data are out of scope and should not be attempted.

How often should the workflow run?
For price monitoring, every 4 to 24 hours is usually enough. For active monitoring of a small set of competitor products, hourly may be justified. Match the cadence to the listing velocity and your rate-limit headroom.

What happens when AliExpress changes its page layout?
The scraper repository may stop returning the expected fields. Your normalization layer will log missing fields, giving you a clear signal of what broke. Update the mapper or switch to a newer repository version when available.

Can this connect to n8n, a CRM, or an AI agent?
Yes. Once the data is in JSONL or loaded into a database, you can feed it into n8n workflows, CRM enrichment tools, or LLM context windows. The data-scrape profile hosts related repositories for business-data and AI-agent workflows.

What should I verify before production use?
Confirm that the repository you chose is currently maintained, that your proxy and session strategy complies with AliExpress policies, that your output schema matches downstream requirements, and that you have monitoring and alerting in place for failures and schema drift.

What's Next

If you want a transparent, maintainable way to collect public AliExpress data, start with the verified repositories:

• data-scrape/aliexpress-scraper for public AliExpress product and seller metadata extraction.
• data-scrape/booking-scraper for adjacent marketplace and travel-data coverage.
• data-scrape for related open-source scraping tools.

For additional public-web-data research context, see the Chinese-language public-web-data guide for e-commerce price monitoring, which covers cross-marketplace research patterns in more depth.

Build the normalization layer first, run it on a small sample, and only scale once the schema and compliance checks are solid.

Comments

[Luis Cruz]

I appreciate the insight into handling dynamic markup and schema drift when scraping AliExpress data. It’s crucial to implement a robust normalization pipeline to ensure consistency, especially given the varying formats and potential for fragmented metadata. One improvement could be to include automated testing for the scraper to catch any potential changes in the HTML structure before they affect data collection. If you're looking for engineering support to enhance this pipeline or tackle any related challenges, I’d be glad to discuss a paid collaboration. How have you approached the issue of maintaining the scraper as the site evolves?