DEV Community: coreclaw

TikTok Scraper GitHub: How to Build a Public Video and Creator Pipeline with Python

coreclaw — Thu, 20 Aug 2026 02:08:40 +0000

TikTok Scraper GitHub: How to Build a Public Video and Creator Pipeline with Python

The most reliable way to turn public TikTok content into a structured research dataset is to combine an open-source scraper repository with a small Python normalization pipeline that you control. This article is for data engineers, growth analysts, and developers who want to collect public video and creator metadata, standardize the fields, and store the results in a format that other tools can consume.

You do not need to build a browser farm or manage proxy rotation from scratch. The data-scrape/tiktok-data-scraper-api and data-scrape/tiktok-video-scraper repositories provide runnable reference implementations. Your job is to wrap them in a pipeline that deduplicates records, enforces a schema, and respects platform terms.

TL;DR

Use an open-source TikTok scraper repository to obtain public video and creator records as JSON.
Normalize the raw JSON into a consistent schema: video URL, creator handle, caption, engagement counts, timestamp, and music info.
Store normalized records as JSONL so downstream tools can stream them without parsing a large array.
Schedule the workflow with cron, a scheduler, or a queue; refresh cadence depends on how fast the content changes.
Always verify current rate limits, robots directives, and TikTok's terms of service before running at scale.
For a Chinese-language public-web-data research reference, see the CoreClaw TikTok scraper guide.

Why TikTok Data Is Hard to Collect at Scale

TikTok pages are heavily rendered in the browser and include anti-bot protections, device fingerprints, and signature parameters. A simple requests.get() call to a video page usually returns JavaScript placeholders instead of the metadata you see in the app.

The common pitfalls are:

Dynamic markup: Video metadata, view counts, and comments load after the initial HTML.
Session signatures: Request URLs often include signatures tied to a session, so a copied URL may stop working quickly.
Rate limiting: Aggressive polling from a single IP triggers blocks or CAPTCHA challenges.
Schema drift: Field names and payload shapes change as the platform evolves.
Proxy overhead: Residential or mobile proxies improve success rates but add cost and operational complexity.

Because of this, most production teams either buy a managed TikTok data API or use an open-source scraper repository and accept the maintenance burden. This article focuses on the second path because it keeps the data flow transparent and the schema under your control.

What the Verified Repositories Provide

The data-scrape/tiktok-data-scraper-api repository is an open-source reference for collecting public TikTok records through a scraper-style API wrapper. It is useful when you want a structured request/response pattern and plan to integrate the scraper into a larger application.

The data-scrape/tiktok-video-scraper repository focuses on extracting public video-level metadata: captions, hashtags, music, engagement statistics, and author information. It is a good fit when your use case centers on content analysis rather than creator monitoring.

Both repositories are starting points, not unlimited services. Read their README files for current setup instructions, dependencies, and any environment variables they require. Do not assume that a repository endpoint shown in an example is still live or unlimited.

Pipeline Design

A maintainable TikTok data pipeline has four stages:

Ingest: Run the scraper repository against a list of public video URLs, creator handles, or hashtags.
Normalize: Map the raw JSON fields into a stable schema regardless of which repository produced them.
Deduplicate: Drop records you already captured in the previous run.
Store and expose: Write JSONL files or load the records into a database, queue, or analytics tool.

This design isolates schema drift to the normalization layer. When TikTok changes a field name, you update one mapper instead of rewriting every downstream query.

Environment Setup

Create a project directory and install the dependencies the repositories list. Most require Python 3.10 or newer plus requests, httpx, or a headless browser driver. For the normalization layer you only need the standard library and a JSON processor.

python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Define environment variables for anything that changes between environments:

export TIKTOK_INPUT_LIST="urls.txt"
export TIKTOK_OUTPUT_DIR="./data"
export TIKTOK_RUN_ID="2026-08-20"

Keep API keys, session tokens, and proxy credentials out of the code. If a repository requires a session cookie or signature helper, load it from the environment and rotate it on the schedule recommended by the repository's documentation.

Runnable Python Normalization Workflow

The script below does not call a live endpoint. It reads representative JSON records that a scraper repository would produce and turns them into a clean, deduplicated JSONL file. Replace the input_path with the actual output directory of your chosen repository.

import json
import os
from datetime import datetime, timezone
from pathlib import Path

INPUT_PATH = os.environ.get("TIKTOK_INPUT_LIST", "sample_tiktok_records.json")
OUTPUT_DIR = Path(os.environ.get("TIKTOK_OUTPUT_DIR", "./data"))
RUN_ID = os.environ.get("TIKTOK_RUN_ID", datetime.now(timezone.utc).strftime("%Y-%m-%d"))

OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
OUTPUT_FILE = OUTPUT_DIR / f"tiktok_normalized_{RUN_ID}.jsonl"
SEEN_FILE = OUTPUT_DIR / "seen_ids.txt"

VIDEO_SCHEMA = {
    "video_url": None,
    "video_id": None,
    "creator_handle": None,
    "creator_display_name": None,
    "caption": None,
    "hashtags": [],
    "music_title": None,
    "music_author": None,
    "view_count": None,
    "like_count": None,
    "comment_count": None,
    "share_count": None,
    "published_at": None,
    "scraped_at": None,
    "source_repo": None,
}


def normalize_video_record(raw: dict, source_repo: str) -> dict:
    """Map a raw repository record to the stable schema."""
    record = VIDEO_SCHEMA.copy()
    record["video_url"] = raw.get("url") or raw.get("video_url") or raw.get("share_url")
    record["video_id"] = raw.get("id") or raw.get("video_id") or raw.get("aweme_id")
    record["creator_handle"] = (
        raw.get("author", {}).get("unique_id")
        or raw.get("creator_handle")
        or raw.get("author_id")
    )
    record["creator_display_name"] = (
        raw.get("author", {}).get("nickname")
        or raw.get("creator_display_name")
    )
    record["caption"] = raw.get("desc") or raw.get("caption") or raw.get("description")
    record["hashtags"] = raw.get("hashtags") or raw.get("text_extra", [])
    record["music_title"] = (
        raw.get("music", {}).get("title")
        or raw.get("music_title")
    )
    record["music_author"] = (
        raw.get("music", {}).get("author")
        or raw.get("music_author")
    )
    record["view_count"] = raw.get("play_count") or raw.get("view_count") or raw.get("stats", {}).get("play_count")
    record["like_count"] = raw.get("digg_count") or raw.get("like_count") or raw.get("stats", {}).get("digg_count")
    record["comment_count"] = raw.get("comment_count") or raw.get("stats", {}).get("comment_count")
    record["share_count"] = raw.get("share_count") or raw.get("stats", {}).get("share_count")
    record["published_at"] = raw.get("create_time") or raw.get("published_at")
    record["scraped_at"] = datetime.now(timezone.utc).isoformat()
    record["source_repo"] = source_repo
    return record


def load_seen_ids(path: Path) -> set:
    if not path.exists():
        return set()
    return set(path.read_text(encoding="utf-8").splitlines())


def save_seen_ids(path: Path, ids: set):
    path.write_text("\n".join(sorted(ids)), encoding="utf-8")


# ---------------------------------------------------------------------------
# Example: load raw records produced by a scraper repository.
# In production, point this at the actual repository output file or queue.
# ---------------------------------------------------------------------------
sample_records = [
    {
        "id": "7280000000000000001",
        "url": "https://www.tiktok.com/@examplecreator/video/7280000000000000001",
        "author": {"unique_id": "examplecreator", "nickname": "Example Creator"},
        "desc": "A short demo video #python #data",
        "hashtags": ["python", "data"],
        "music": {"title": "Demo Track", "author": "Demo Artist"},
        "stats": {"play_count": 15400, "digg_count": 920, "comment_count": 45, "share_count": 12},
        "create_time": "2026-08-18T14:30:00+00:00",
    }
]

seen_ids = load_seen_ids(SEEN_FILE)
new_records = []

for raw in sample_records:
    record = normalize_video_record(raw, source_repo="tiktok-video-scraper")
    vid = record.get("video_id")
    if vid and vid in seen_ids:
        continue
    if vid:
        seen_ids.add(vid)
    new_records.append(record)

with open(OUTPUT_FILE, "w", encoding="utf-8") as f:
    for record in new_records:
        f.write(json.dumps(record, ensure_ascii=False) + "\n")

save_seen_ids(SEEN_FILE, seen_ids)

print(f"Wrote {len(new_records)} new records to {OUTPUT_FILE}")
print(f"Total unique video IDs tracked: {len(seen_ids)}")

The mapper uses fallback chains for every field. If a repository renames desc to caption, the script still captures the value. This is the layer that saves you from nightly breakage.

Representative Output

After normalization, each line in tiktok_normalized_2026-08-20.jsonl looks like this:

{
  "video_url": "https://www.tiktok.com/@examplecreator/video/7280000000000000001",
  "video_id": "7280000000000000001",
  "creator_handle": "examplecreator",
  "creator_display_name": "Example Creator",
  "caption": "A short demo video #python #data",
  "hashtags": ["python", "data"],
  "music_title": "Demo Track",
  "music_author": "Demo Artist",
  "view_count": 15400,
  "like_count": 920,
  "comment_count": 45,
  "share_count": 12,
  "published_at": "2026-08-18T14:30:00+00:00",
  "scraped_at": "2026-08-20T10:15:00+00:00",
  "source_repo": "tiktok-video-scraper"
}

JSONL is a good default because you can append new records without rewriting the whole file, stream it into pandas or DuckDB, and load it into a queue such as RabbitMQ or SQS with minimal parsing overhead.

Build vs. Buy Checklist

Dimension	Open-source scraper repository	Managed TikTok data API
Best for	Teams that need full schema control and can maintain the scraper	Teams that need data immediately without infrastructure work
Setup	Clone repo, install deps, configure proxies/sessions	Sign up, copy endpoint, set API key
Data coverage	Depends on the repository and current site structure	Depends on provider coverage; confirm before buying
Output format	Raw or custom normalized JSON/JSONL	Usually structured JSON with fixed schema
Maintenance burden	High: updates needed when site markup changes	Low to medium: provider handles breakage
Integration path	Local scripts, cron, queues, custom API wrapper	Direct HTTP API, SDK if available
Rate/freshness	Set by your proxy and session strategy	Set by provider plan; verify current limits
Pricing	Infrastructure cost only; proxies are usually paid	Per-request or per-result; confirm on official pricing page

There is no universally better choice. A solo analyst with one hashtag to track may be fine with an open-source repository. A marketing agency monitoring thousands of creators per day will usually prefer a managed API once the cost of maintenance exceeds the subscription price.

Business Use Cases

Public TikTok video and creator data supports several workflows:

Trend detection: Track hashtag velocity and sound usage across a cohort of videos.
Creator research: Build a database of creators in a niche, then filter by engagement rate, follower range, or posting cadence.
Competitor monitoring: Watch how often a brand posts, which sounds it uses, and how engagement changes over time.
Content strategy: Identify caption patterns, posting times, and hashtag clusters associated with high-performing videos.
Influencer vetting: Compare public engagement metrics against claimed reach before signing a contract.

In every case, stay within the boundaries of public data. Do not attempt to access private accounts, direct messages, or data that requires authentication beyond what the account owner has made public.

Compliance and Maintenance Notes

TikTok's terms of service, robots directives, and regional privacy laws apply regardless of whether you use an open-source tool or a paid API. Before running any scraper at scale:

Read the current TikTok Terms of Service and robots.txt behavior.
Collect only public data. Private accounts, non-public metrics, and personal information of non-public figures are out of scope.
Respect rate limits. Add backoff, jitter, and retry logic. Do not hammer the platform from a single IP.
Keep a log of what you collected, when, and from which public URL. This audit trail matters for compliance and debugging.
Rotate session credentials and proxies on the schedule recommended by the repository documentation.
Monitor schema drift. Schedule a small weekly job that alerts you when expected fields disappear or change type.

A scraper is never maintenance-free. Plan for at least a few hours per month of upkeep, more if TikTok makes significant changes to its page structure.

FAQ

Is there an official TikTok API for this data?
TikTok offers official APIs for authorized partners and certain research programs, but access is restricted and requires approval. The repository-based workflow described here is for public web data that does not rely on official API access.

What data fields are returned?
The exact fields depend on the repository and the current page structure. Common fields include video URL, video ID, creator handle, caption, hashtags, music info, view count, like count, comment count, share count, and publish timestamp. Always inspect a sample response before building downstream logic.

Can I use this for private accounts or direct messages?
No. This workflow is for public videos and public creator profiles only. Private data, direct messages, and account-cracking techniques are not covered and should not be attempted.

How often should the workflow run?
For trend tracking, every 4–24 hours is usually enough. For live campaign monitoring, hourly may be justified. Match the cadence to the content velocity and your rate-limit headroom.

What happens when TikTok changes its page layout?
The scraper repository may stop returning the expected fields. Your normalization layer will log missing fields, giving you a clear signal of what broke. Update the mapper or switch to a newer repository version when available.

Can this connect to n8n, a CRM, or an AI agent?
Yes. Once the data is in JSONL or loaded into a database, you can feed it into n8n workflows, CRM enrichment tools, or LLM context windows. The data-scrape profile hosts related repositories for business-data and AI-agent workflows.

What should I verify before production use?
Confirm that the repository you chose is currently maintained, that your proxy and session strategy complies with TikTok's policies, that your output schema matches downstream requirements, and that you have monitoring and alerting in place for failures and schema drift.

What's Next

If you want a transparent, maintainable way to collect public TikTok data, start with the verified repositories:

data-scrape/tiktok-data-scraper-api for a scraper-style API reference.
data-scrape/tiktok-video-scraper for public video metadata extraction.
data-scrape for related open-source scraping tools.

For additional public-web-data research context, see the CoreClaw TikTok scraper guide (Chinese-language).

Build the normalization layer first, run it on a small sample, and only scale once the schema and compliance checks are solid.

Build vs Buy Web Scraping: When to Use a Scraper API Instead of Self-Hosting

coreclaw — Thu, 20 Aug 2026 01:07:06 +0000

Build vs Buy Web Scraping: When to Use a Scraper API Instead of Self-Hosting

The build-vs-buy decision for web scraping comes down to total cost of ownership, maintenance burden, and how quickly your team needs structured data. For most organizations — from solo developers to enterprise data teams — a managed scraper API or platform delivers clean, structured web data faster and more reliably than self-hosted infrastructure. Build your own only when you need fine-grained control over request behavior, custom browser fingerprints, or niche target sites that no platform covers.

This article is for engineering managers, data teams, and developers deciding whether to invest engineering time in a custom scraping stack or to use a managed platform that handles proxies, anti-bot detection, and output formatting.

TL;DR

Build when you have unique target sites, specialized parsing needs, full-time scraping engineers, or strict data-residency requirements.
Buy when you need data from common sources (Google, Amazon, LinkedIn, social media), want structured JSON instead of raw HTML, and value time-to-data over infrastructure ownership.
Hybrid works well: use a managed platform for common data sources and keep custom scripts for edge cases.

Why Building Your Own Scraper Is Harder Than It Looks

A self-hosted scraping pipeline looks deceptively simple: send an HTTP request, parse the response, store the data. In production, the actual maintenance surface is much larger:

Proxy management. Rotating residential and datacenter proxies, health checks, geo-targeting, and cost optimization become a full-time job at scale.
Anti-bot detection. Cloudflare, PerimeterX, DataDome, and similar systems fingerprint browsers and block IP ranges. Keeping up with detection updates is continuous work.
Browser automation. Headless browser orchestration with Playwright or Puppeteer adds memory, timeout, and crash-handling complexity.
Parser maintenance. Target sites change their DOM structure regularly. Each change breaks CSS selectors or XPath expressions and requires a manual fix.
Scaling. Concurrency limits, rate limiting, retry logic, queue management, and result storage all need dedicated infrastructure.
Monitoring. You need alerting for failures, latency, data quality drift, and proxy health — not just "did the job run."
Compliance review. robots.txt adherence, terms-of-service review, rate-limiting policies, and data retention practices need periodic legal or compliance review.

A team that builds all of this from scratch typically spends most of its time on infrastructure maintenance rather than data analysis.

Build vs Buy: Side-by-Side Comparison

Dimension	Self-Hosted Scraper	Managed Scraper API / Platform
Setup time	Days to weeks	Minutes (API key and first request)
Proxy management	You manage rotation, health, cost	Included, no configuration needed
Anti-bot handling	You maintain workarounds	Platform handles updates
Output format	Raw HTML or custom-parsed JSON	Structured JSON with typed fields
Scaling	You provision servers, queues, storage	Platform scales on demand
Maintenance burden	High — parser fixes, proxy updates, monitoring	Low — platform maintains parsers
Cost model	Fixed infra plus variable proxy costs	Pay-per-result or subscription
Customization	Full control over headers, timing, parsing	Limited to platform's supported sources
Compliance	You own all legal and ToS review	Platform provides public-data-only guardrails

The comparison is not "managed is always better." It is "managed is better for common data sources at scale; self-hosted is better for niche or proprietary pipelines."

When to Build Your Own Scraper

Build when at least one of these is true:

Niche or proprietary target sites. If your targets are small, specialized, or region-specific sites that no platform covers, a custom scraper is your only option.
Custom browser automation. If you need to simulate complex user interactions — login flows, multi-step forms, dynamic SPAs — a custom script gives you full control.
Full-time scraping engineers. If your team has dedicated scraping specialists, the marginal cost of maintaining infrastructure is lower.
Strict data residency. If data must stay within a specific region or network, self-hosting may be the only compliant option.
Research or one-off projects. For a one-time data pull, a quick script may be faster than onboarding to a platform.

When to Buy a Managed Scraper API

Buy when any of these apply:

Common data sources. Google Maps, Amazon, LinkedIn, Instagram, YouTube, TikTok, and Google Search are covered by managed platforms. Building custom scrapers for these is reinventing a wheel that many teams maintain full-time.
Production workloads. If data feeds a product, dashboard, or AI agent, reliability matters more than owning the infrastructure.
Structured JSON, not HTML. If your downstream consumers expect typed fields — name, price, rating, address — a platform that returns structured JSON saves parsing work.
No dedicated scraping team. Most engineering teams do not have full-time scraping specialists. A managed platform fills that gap.
Fast time-to-data. If the business needs data this week, not next quarter, a managed API is the fastest path.

Runnable Example: Managed API vs Raw Requests

The following Python example shows how the same task — fetching structured product data — compares between a self-hosted approach and a managed API. All secrets and endpoints use environment variables; never hardcode credentials.

import os
import json
import requests
from bs4 import BeautifulSoup

# ── Self-hosted approach: parse raw HTML yourself ──────────────
def scrape_product_self_hosted(url: str) -> dict:
    """
    Fetch a product page and parse fields manually.
    Requires your own proxy, retry logic, and parser maintenance.
    """
    proxy = os.environ.get("SELF_PROXY_URL", "")
    headers = {
        "User-Agent": os.environ.get(
            "USER_AGENT", "Mozilla/5.0 (compatible; MyScraper/1.0)"
        ),
    }
    proxies = {"http": proxy, "https": proxy} if proxy else None

    resp = requests.get(url, headers=headers, proxies=proxies, timeout=30)
    resp.raise_for_status()

    soup = BeautifulSoup(resp.text, "html.parser")
    # These selectors WILL break when the site changes layout.
    title_el = soup.select_one("h1#title")
    price_el = soup.select_one("span#priceblock_ourprice")
    rating_el = soup.select_one("span#acrPopover")

    return {
        "title": title_el.get_text(strip=True) if title_el else None,
        "price": price_el.get_text(strip=True) if price_el else None,
        "rating": rating_el.get_text(strip=True) if rating_el else None,
        "source_url": url,
    }


# ── Managed API approach: structured JSON in one call ──────────
def scrape_product_managed_api(url: str) -> dict:
    """
    Call a managed scraper API that handles proxies, anti-bot,
    and returns structured JSON. Replace ENDPOINT with the
    current value from your platform console.
    """
    api_key = os.environ["SCRAPER_API_KEY"]
    endpoint = os.environ["SCRAPER_API_ENDPOINT"]  # from console
    payload = {"url": url, "output_format": "json"}
    headers = {"Authorization": f"Bearer {api_key}"}

    resp = requests.post(endpoint, json=payload, headers=headers, timeout=60)
    resp.raise_for_status()
    return resp.json()


# ── Run both for comparison ────────────────────────────────────
if __name__ == "__main__":
    target = os.environ.get("TARGET_URL", "https://example.com/product/123")

    print("Self-hosted result:")
    print(json.dumps(scrape_product_self_hosted(target), indent=2))

    print("\nManaged API result:")
    print(json.dumps(scrape_product_managed_api(target), indent=2))

What this example shows

The self-hosted path requires proxy configuration, user-agent management, HTML parsing, and selector maintenance.
The managed path delegates all of that to the platform and returns structured JSON.
Both approaches use environment variables — no hardcoded secrets or endpoints.

Output Checklist: What to Expect from a Managed Platform

Before committing to a managed scraping platform, verify these output qualities:

[ ] Returns structured JSON with documented field names
[ ] Includes source URL for traceability
[ ] Handles pagination and result limits
[ ] Returns error responses with actionable messages
[ ] Supports the geographic regions you need
[ ] Provides rate-limit and quota information in headers or response body
[ ] Maintains parsers when target sites change layout

Business Use Cases by Approach

Use Case	Recommended Approach	Why
Google Maps lead generation	Managed API	Covered by platforms, structured fields, scale
Amazon price monitoring	Managed API	Frequent layout changes, anti-bot complexity
LinkedIn B2B enrichment	Managed API	High anti-bot, login-walled content
Instagram/YouTube creator research	Managed API	Platform-level parsers, engagement fields
Google SERP rank tracking	Managed API	Geographic targeting, structured results
Niche forum or proprietary site	Self-hosted	No platform coverage, custom parsing needed
Internal tool behind a VPN	Self-hosted	Network access constraints
Multi-step form interaction	Self-hosted	Complex browser automation

For teams that decide a managed platform fits their needs, the CoreClaw pricing page shows the cost model, and the CoreClaw Workers Store lists ready-made scrapers for common data sources. If you want to deploy a custom scraper on managed infrastructure, the CoreClaw console lets you build and deploy without managing servers.

Limitations and Compliance

No approach removes compliance responsibility. Whether you build or buy, you must respect target-site terms of service, applicable laws (including privacy regulations), and robots directives where relevant.
Managed platforms scope to public data. Reputable platforms focus on publicly accessible web data and do not help with credential abuse, private data, or bypassing access controls.
Self-hosted scrapers need legal review. If you build your own, you own all compliance review — rate limits, data retention, and terms-of-service analysis.
Freshness varies by source. Some data (prices, SERP results) changes daily; other data (business addresses) changes rarely. Match refresh frequency to the data's natural volatility.
Platforms are not unlimited. Quotas, rate limits, and geographic coverage vary. Verify current limits on the provider's official documentation or CoreClaw pricing page before committing to production volumes.

For a broader discussion of what constitutes public web data and the compliance considerations that apply to both build and buy paths, this Chinese-language public web data compliance guide covers the topic in depth.

FAQ

1. Is building a web scraper cheaper than using an API?

It depends on scale and team composition. At low volume with a small team, a quick script may be cheaper. At production volume, the total cost of proxy infrastructure, maintenance hours, and infrastructure monitoring often exceeds the per-result cost of a managed API.

2. What maintenance does a self-hosted scraper require?

Parser fixes when target sites change layout, proxy rotation and health monitoring, anti-bot workaround updates, retry logic, and data quality checks. Expect most engineering time to go to maintenance rather than new features.

3. Can I use both approaches together?

Yes. A common hybrid pattern is to use a managed platform for common data sources (Google Maps, Amazon, LinkedIn) and keep custom scripts for niche sites or proprietary pipelines.

4. How do managed platforms handle proxies and anti-bot?

Managed platforms maintain proxy pools, rotate IPs automatically, and update anti-bot strategies as detection systems evolve. The user does not configure proxies or CAPTCHA solutions.

5. What should I check before choosing a scraping platform?

Verify the data sources covered, output format, pricing model, geographic coverage, rate limits, documentation quality, and whether the platform scopes to public data only.

6. Do I need residential proxies if I use a managed scraper?

No. A managed platform handles proxy management internally. If you self-host, residential proxies may be necessary for targets that block datacenter IPs, but you then own the cost and rotation logic.

7. Can a managed API feed data to my AI agent or automation pipeline?

Yes. Managed scraping APIs return JSON that can be passed directly to AI agents, CRM systems, analytics dashboards, or automation tools. The structured format eliminates the parsing work needed with raw HTML.

Next Steps

If you are evaluating the buy path:

Browse the CoreClaw Workers Store for ready-made scrapers covering Google Maps, Amazon, LinkedIn, Instagram, YouTube, and more.
Review the CoreClaw pricing page to understand the cost model.
Use the CoreClaw console to deploy a custom scraper on managed infrastructure if your use case needs a specific source.

If you are evaluating the build path, document your total cost of ownership — including proxy costs, maintenance hours, and infrastructure overhead — before committing to self-hosting.

X Scraper GitHub: How to Extract Public Posts and Profiles with Python

coreclaw — Wed, 19 Aug 2026 02:03:47 +0000

X Scraper GitHub: How to Extract Public Posts and Profiles with Python

If you need a reproducible way to convert public X (formerly Twitter) posts and profiles into a structured dataset, the most practical starting point is an X scraper GitHub project that ships example code, a normalization layer, and a documented output schema. This article is for developers, data analysts, and research teams who want to build a public-post monitor from open-source Python rather than maintaining a full browser-automation stack themselves.

Quick answer

A repository-based X/Twitter scraper is a Python project that:

Loads a public post or profile URL and returns a normalized JSON record.
Keeps source URL and collection timestamp on every record so you can audit results later.
Lets you chain extraction, normalization, and storage as separate, testable steps.

The data-scrape GitHub organization hosts two relevant reference repositories for this workflow: the X Scraper project and the companion Twitter Scraper repository. Treat both as implementation references, not guaranteed production services. Always read the README, run the example with a small input set, and verify the output before relying on any field, endpoint, or claim.

Why public X/Twitter data is harder to monitor than it looks

Anyone building an X/Twitter monitor quickly runs into the same five constraints:

Dynamic rendering. Post bodies, media URLs, and profile metadata are loaded by JavaScript after the initial HTML response. A plain HTTP client often receives an empty shell.
Layout drift. CSS selectors, embedded JSON paths, and component structure change without warning. Extraction logic that worked last week may return null fields this week.
Rate limiting and session behavior. Repeated requests from the same IP or user-agent pattern can trigger throttling, empty responses, or interactive challenges.
Quota and access tiers. The official X API has tiered pricing and rate limits. A repository-based workflow is appealing precisely because it does not depend on an API key, but it inherits the anti-automation surface area of public web pages.
Identity and duplication. The same post can be re-shared, the same handle can change display name, and retweet counts update independently of post body. Any monitor needs stable keys and de-duplication rules.

These constraints make self-hosted scraping viable for prototypes but expensive to maintain at scale. A repository-based workflow gives you a middle ground: you run the code, control the schema, and decide how much infrastructure to own.

What an X/Twitter scraper returns

A "scraper" in this context is a wrapper around the extraction layer. You pass a public post URL, a profile URL, or a search query, and the scraper returns normalized JSON. The exact fields depend on the repository implementation and the page type, but typical record shapes include:

Post-level fields

post_id or tweet identifier
author_handle and author_display_name
post_text (the body text, cleaned)
created_at (page-visible timestamp)
language (when available)
reply_count, repost_count, like_count, quote_count, view_count (page-visible counts)
media_urls (first image or video URL when present)
post_url (the canonical source URL)

Profile-level fields

handle, display_name, bio
follower_count, following_count
post_count (tweets and replies combined)
verified, profile_image_url
profile_url (the canonical source URL)

Provenance

collected_at: the time your workflow observed the page
source: the public page origin
review_status: a label for human validation

Because public pages differ by region, login state, and whether the account is verified or restricted, treat any field as optional until you have validated it against a controlled sample.

Step-by-step: building a public-post monitor

The workflow below assumes you have installed one of the repository-based tools and configured an endpoint or local runner according to its documentation. It uses environment variables for anything account-specific.

1. Set environment variables

export X_SCRAPER_ENDPOINT="https://your-endpoint-or-localhost.example.com/scrape"
export X_SCRAPER_API_KEY="your_api_key_if_required"
export X_OUTPUT_DIR="./data"

Replace the endpoint with the current value shown in the repository README or your own deployment.

2. Request a single public post

import json
import os
from datetime import datetime, timezone
from pathlib import Path

import requests

API_ENDPOINT = os.environ["X_SCRAPER_ENDPOINT"]
API_KEY = os.environ.get("X_SCRAPER_API_KEY")
OUTPUT_DIR = Path(os.environ.get("X_OUTPUT_DIR", "./data"))
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

headers = {"Content-Type": "application/json"}
if API_KEY:
    headers["Authorization"] = f"Bearer {API_KEY}"

# Example payload; confirm the exact schema in the repository README.
payload = {
    "url": "https://x.com/verge/status/1234567890123456789",
    "include_replies": False,
}

response = requests.post(API_ENDPOINT, headers=headers, json=payload, timeout=120)
response.raise_for_status()
raw_post = response.json()
print(json.dumps(raw_post, indent=2, ensure_ascii=False)[:1000])

3. Normalize the post record

def clean_text(value):
    if not isinstance(value, str):
        return None
    value = " ".join(value.split())
    return value or None


def as_positive_int(value):
    if isinstance(value, bool):
        return None
    if isinstance(value, int) and value > 0:
        return value
    if isinstance(value, str):
        digits = "".join(c for c in value if c.isdigit())
        if digits:
            return int(digits)
    return None


def normalize_post(raw, collected_at):
    post_id = clean_text(raw.get("post_id") or raw.get("id_str") or raw.get("id"))
    handle = clean_text(raw.get("author_handle") or raw.get("user_screen_name"))

    if not post_id or not handle:
        return None

    record = {
        "post_id": post_id,
        "author_handle": handle,
        "author_display_name": clean_text(
            raw.get("author_display_name") or raw.get("user_name")
        ),
        "post_text": clean_text(raw.get("post_text") or raw.get("text") or raw.get("full_text")),
        "created_at": clean_text(raw.get("created_at")),
        "language": clean_text(raw.get("lang")),
        "reply_count": as_positive_int(raw.get("reply_count")),
        "repost_count": as_positive_int(raw.get("repost_count") or raw.get("retweet_count")),
        "like_count": as_positive_int(raw.get("like_count") or raw.get("favorite_count")),
        "quote_count": as_positive_int(raw.get("quote_count")),
        "view_count": as_positive_int(raw.get("view_count")),
        "media_urls": raw.get("media_urls") or [],
        "post_url": clean_text(raw.get("post_url") or raw.get("url")),
        "collected_at": collected_at,
        "source": "public X post page",
        "review_status": "needs_review",
    }
    return record


collected_at = datetime.now(timezone.utc).isoformat()
records = []
post = normalize_post(raw_post, collected_at)
if post:
    records.append(post)

output_file = OUTPUT_DIR / f"post_{datetime.now(timezone.utc).strftime('%Y%m%d_%H%M%S')}.json"
output_file.write_text(json.dumps(records, indent=2, ensure_ascii=False), encoding="utf-8")
print(f"Saved {len(records)} normalized record(s) to {output_file}")

4. Normalize a profile record

def normalize_profile(raw, collected_at):
    handle = clean_text(raw.get("handle") or raw.get("screen_name"))
    if not handle:
        return None

    return {
        "handle": handle,
        "display_name": clean_text(raw.get("display_name") or raw.get("name")),
        "bio": clean_text(raw.get("bio") or raw.get("description")),
        "follower_count": as_positive_int(raw.get("follower_count")),
        "following_count": as_positive_int(raw.get("following_count")),
        "post_count": as_positive_int(raw.get("post_count") or raw.get("statuses_count")),
        "verified": bool(raw.get("verified")),
        "profile_image_url": clean_text(raw.get("profile_image_url")),
        "profile_url": clean_text(raw.get("profile_url") or f"https://x.com/{handle}"),
        "collected_at": collected_at,
        "source": "public X profile page",
        "review_status": "needs_review",
    }

This separation keeps extraction, normalization, and storage independent. If the extraction layer changes, you can update one function without rewriting the monitor logic.

Representative output shape

After normalization, a single post record looks like this:

{
  "post_id": "1234567890123456789",
  "author_handle": "verge",
  "author_display_name": "The Verge",
  "post_text": "Apple unveils new on-device language model for iOS 19.",
  "created_at": "2026-08-12T14:22:00.000Z",
  "language": "en",
  "reply_count": 84,
  "repost_count": 312,
  "like_count": 1502,
  "quote_count": 27,
  "view_count": 189000,
  "media_urls": [
    "https://pbs.twimg.com/media/Example123.jpg"
  ],
  "post_url": "https://x.com/verge/status/1234567890123456789",
  "collected_at": "2026-08-19T02:00:00+00:00",
  "source": "public X post page",
  "review_status": "needs_review"
}

A null field means the value was not present or not parseable in that run, not that it is zero or empty. Keeping that distinction visible prevents bad downstream decisions.

Use cases for a public-post monitor

News and trend tracking. Capture posts from a curated list of newsroom and analyst handles, then surface terms that spike in frequency or velocity.

Brand and reputation monitoring. Watch your brand name, product names, and competitor names, then route posts with high engagement to a human review queue.

Market research. Build a structured corpus of public posts around a topic for content analysis, sentiment scoring, or training data preparation.

Academic and policy research. Track a small set of public handles and hashtags with explicit consent and ethics review, then archive snapshots for later analysis.

Operations alerts. Notify a security or trust-and-safety team when specific keywords, handles, or media signatures appear on public posts.

Each use case should start with a narrow scope, clear data retention rules, and a manual review gate before any financial, legal, or personnel decision.

Repository workflow versus building from scratch

Dimension	Repository-based scraper	Build your own from scratch
Setup time	Hours to days, depending on the repo	Days to weeks for a reliable prototype
Maintenance	You update selectors and dependencies	You own the full extraction and anti-detection stack
Schema control	You define the normalization layer	You define everything
Proxy/headless overhead	Depends on repo design; may still need proxies	You manage browsers, proxies, and rotation
Field certainty	Validate from actual test outputs	Validate from your own parsing logic
Compliance	Your responsibility either way	Your responsibility either way

Neither option removes the obligation to respect X's terms, applicable law, and privacy requirements. A repository gives you a head start on code; it does not give you permission to collect or use data indiscriminately.

For a deeper look at structuring public-data projects and pipeline design choices that work across sources, see the Chinese-language public web data market research guide — it covers source selection, normalization, and the limits teams usually discover after the first pilot.

Limitations and compliance

No official X API guarantee. Public pages are not a stable data contract. Fields, URLs, and availability can change.
Regional variation. The same post may render differently depending on location, device fingerprint, and whether the viewer is logged in.
Rate and volume limits. Running a monitor too aggressively can trigger blocks. Start with a small cadence and increase only after observing stable behavior.
Data accuracy. Public posts can be edited, deleted, or restricted after you collect them. Always validate high-impact records before acting.
Legal and platform terms. Use public data only for lawful purposes and in compliance with X's terms of service, robots directives where applicable, and applicable privacy laws. Do not evade access controls, logins, or rate limits, and do not collect data behind authentication walls.
Personal data. If your records contain personal data, define retention, deletion, and access-control rules before you start collecting.

FAQ

Is the repository a managed production API?

No. The X Scraper repository and the Twitter Scraper repository are code references. Review their READMEs, dependencies, and recent commits before using them in any system.

What fields are guaranteed?

None. Public page extraction depends on the current page structure, region, login state, and account type. Validate every field you plan to use with a controlled test sample.

How often should the monitor run?

Start hourly for a small handle set, then move to daily for broader coverage. Increase frequency only if the decision you are supporting actually benefits from faster updates and you can handle the increased block risk.

Can this feed into a CRM, dashboard, or AI agent?

Yes, but only after normalization and review. The JSON output in the example above is designed to be stored in a database, sent to a webhook, or loaded as context for an AI agent. Keep provenance fields so the downstream system knows where each value came from.

What happens when X changes its page layout?

The extraction step may break or return partial records. A well-structured monitor detects increased null rates or parse failures and alerts a human to inspect the repository for updates.

Should I keep every raw response?

Only retain what your use case and governance rules justify. Keep enough to audit a result, but do not accumulate unnecessary personal or sensitive content.

Where do I inspect the referenced projects?

Start with the data-scrape GitHub profile, then review the X Scraper repository and the Twitter Scraper repository.

Next step

Before running a production monitor, define the exact handles, hashtags, or search queries you need, write a target JSON schema, collect a small test sample, and assign someone to validate the output. The repositories above are useful starting points for that evaluation; the durable asset is the documented, reviewable workflow your team builds around them.

No-Code Web Scraping Tools: How to Choose a Ready-Made Scraper Marketplace

coreclaw — Mon, 17 Aug 2026 01:42:15 +0000

No-Code Web Scraping Tools: How to Choose a Ready-Made Scraper Marketplace

A no-code web scraping tool built on a hosted, ready-made scraper marketplace lets you send one API call and receive structured JSON in return, without managing proxies, headless browsers, or parser maintenance. If your bottleneck is data, not infrastructure, the right next step is to compare marketplaces on the dimensions that actually drive cost and reliability — coverage, output schema, pricing model, delivery mechanism, and compliance posture — and run a small test job before you commit your team's pipeline.

This article walks through what a scraper marketplace is, the criteria that separate the strong ones from the brittle ones, a runnable Python workflow that talks to a marketplace endpoint, and the trade-offs you should expect before going to production.

TL;DR

A ready-made scraper marketplace is a hosted catalog of pre-built scrapers ("actors" or "workers") that return structured data when you submit a job.
The right pick depends on six criteria: data coverage, output schema stability, pricing model, delivery mechanism (sync vs polling vs webhook), maintenance burden, and compliance posture.
You can evaluate a marketplace in one afternoon by submitting three small test jobs on real sources you care about.
For a marketplace that bundles proxy rotation, anti-detection, and a pay-per-result model, browse the CoreClaw product store for ready-made workers you can run on demand.

Why Custom Scrapers Hit a Wall

Most engineering teams that try to build scrapers in-house eventually hit the same five walls:

Proxy management. Public sites block requests from data-center IP ranges within minutes. You either rotate residential proxies (cost) or build retry logic that throttles throughput (time).
Anti-bot detection. Fingerprinting, TLS checks, canvas hashing, and behavioral scoring are now the default on any site with a monetization motive.
Schema drift. Layouts change. The day after launch, your selector stops matching and your pipeline silently returns empty rows.
Regional variation. A page that works from a US egress IP may render differently from a Frankfurt egress IP, with different fields visible.
Operational drag. Every page that breaks is a ticket. Every region you add is a new proxy pool. Every schema change is a hotfix.

The result is that "a small scraping task" quietly turns into a maintenance department. A no-code web scraping tool built on a marketplace moves that burden to a provider that runs scraping infrastructure at scale, so your team keeps writing the application logic that uses the data.

What Is a Ready-Made Scraper Marketplace?

A scraper marketplace is a hosted catalog of pre-built data extractors, each one tuned for a particular site or data source. You typically interact with it in three ways:

Pick a worker (or "actor"). Each entry in the catalog targets one data source: Google Maps, Amazon product pages, Google Search results, Instagram posts, YouTube channels, and so on.
Submit a job. You pass input parameters (a query, a region, a list of URLs) and a configuration (output format, delivery method, paging rules).
Receive structured data. The marketplace returns JSON or CSV records that include the fields the worker extracts — names, addresses, prices, reviews, comments, video metadata, and so on.

The marketplace operator owns the proxies, headless browsers, anti-detection logic, parser updates, and runtime. You own the input, the post-processing, and the destination system.

The marketplace model is different from a self-hosted open-source scraper (you own everything), a single-purpose SaaS API (one endpoint, one data source), and a custom build (you write the worker). It is closest to the "API for many APIs" idea: one integration covers many data sources.

Six Criteria That Matter When You Compare Marketplaces

The marketing pages look similar, so evaluate each marketplace against the same six dimensions. Skim the table below, then read the details.

1. Data coverage

Coverage means the breadth of sources a marketplace supports out of the box, and the depth each source covers. A marketplace that lists "100+ workers" but only exposes surface-level fields for each one is not the same as a marketplace with fewer workers and richer output schemas.

When you compare coverage:

List the data sources you actually need. If three sources cover 90% of your work, prioritize those.
Inspect a sample response. Are the fields you need present, or only the obvious ones?
Check how regional variants are handled. A worker that works for the US Google SERP may not work for the DE SERP.

2. Output schema stability

A worker that returns a slightly different schema every release will break your downstream code. Look for:

A documented schema per worker, ideally versioned.
A changelog or migration guide for breaking updates.
The ability to pin a worker to a specific version.

3. Pricing model

There are three common pricing models:

Pay-per-result. You pay for each delivered record. Cost is transparent and scales with usage.
Subscription. You pay a flat fee for a quota of records per month. Cost is predictable but front-loaded.
Compute-hour. You pay for runtime regardless of result count. Cost can spike on slow or retry-heavy jobs.

For variable workloads, pay-per-result is usually the simplest. For steady workloads, subscriptions can be cheaper. Always confirm the current pricing on the official page; do not assume the figures you read in third-party reviews.

4. Delivery mechanism

How does the marketplace return your data? Three patterns are common:

Synchronous. The job runs and returns the result in a single HTTP call. Best for small jobs.
Polling. You submit, get a job ID, and poll until the job is complete. Best for medium jobs.
Webhook. You submit with a callback URL and receive the result later. Best for batch jobs.

The right choice depends on your job size and your client's tolerance for long-running HTTP calls.

5. Maintenance and SLA

Look for:

Status pages or incident reports.
A documented change policy for worker schemas.
Support for pinning worker versions.
A real SLA if you run production workloads.

6. Compliance posture

A marketplace operator should publish:

The data sources it collects from.
Its policy on terms-of-service compliance for those sources.
Its policy on PII handling and retention.
Whether it supports regional data residency.

If a marketplace is silent on these, assume the worst and ask directly.

A Runnable Python Workflow

The example below submits a job to a marketplace endpoint, polls for completion, and saves the result to a JSON file. The endpoint and token come from environment variables so you never hardcode credentials or invent a URL.

import json
import os
import time
from typing import Any

import requests

# Configuration from the environment. Replace with the values from your account.
API_TOKEN = os.environ["MARKETPLACE_API_TOKEN"]
ENDPOINT = os.environ["MARKETPLACE_ENDPOINT"]
POLL_INTERVAL = int(os.environ.get("MARKETPLACE_POLL_SECONDS", "10"))
OUTPUT_PATH = os.environ.get("MARKETPLACE_OUTPUT", "marketplace_results.json")


def submit_job(endpoint: str, token: str, payload: dict[str, Any]) -> str:
    """Submit a job and return the job ID for polling."""
    response = requests.post(
        endpoint,
        headers={
            "Authorization": f"Bearer {token}",
            "Content-Type": "application/json",
        },
        json=payload,
        timeout=60,
    )
    response.raise_for_status()
    job_id = response.json().get("job_id")
    if not job_id:
        raise RuntimeError("No job_id returned. Check the endpoint and credentials.")
    return job_id


def fetch_results(endpoint: str, token: str, job_id: str) -> list[dict[str, Any]]:
    """Poll the job until it completes and return the result records."""
    status_url = f"{endpoint.rstrip('/')}/{job_id}"
    while True:
        response = requests.get(
            status_url,
            headers={"Authorization": f"Bearer {token}"},
            timeout=60,
        )
        response.raise_for_status()
        data = response.json()
        status = data.get("status")
        if status == "completed":
            return data.get("results", [])
        if status in {"failed", "error", "canceled"}:
            raise RuntimeError(f"Job did not succeed: {status} ({data.get('message')})")
        time.sleep(POLL_INTERVAL)


def save_records(records: list[dict[str, Any]], path: str) -> None:
    with open(path, "w", encoding="utf-8") as f:
        json.dump(records, f, indent=2, ensure_ascii=False)


def main() -> None:
    # Example payload: a generic product lookup. Replace with the worker
    # name and inputs documented by your marketplace provider.
    payload = {
        "worker": "amazon_global_product_scraper",
        "input": {
            "queries": ["wireless earbuds", "office chair"],
            "marketplace": "US",
        },
        "output_format": "json",
    }

    print("Submitting marketplace job...")
    job_id = submit_job(ENDPOINT, API_TOKEN, payload)
    print(f"Job submitted: {job_id}. Polling for completion...")

    records = fetch_results(ENDPOINT, API_TOKEN, job_id)
    print(f"Retrieved {len(records)} records.")

    save_records(records, OUTPUT_PATH)
    print(f"Saved to {OUTPUT_PATH}")


if __name__ == "__main__":
    main()

Run it with:

export MARKETPLACE_API_TOKEN="your_token_here"
export MARKETPLACE_ENDPOINT="https://console.coreclaw.com/api/v1/jobs"
export MARKETPLACE_POLL_SECONDS="10"
export MARKETPLACE_OUTPUT="products.json"
python marketplace_job.py

The endpoint path depends on the worker you run and the conventions of the provider you choose. Copy the current endpoint from your CoreClaw worker settings or the deploy page rather than guessing a URL.

Representative Output

A single record from a product worker might look like this:

{
  "title": "Acme Wireless Earbuds Pro",
  "asin": "B0EXAMPLE001",
  "price": 49.99,
  "currency": "USD",
  "rating": 4.4,
  "review_count": 1284,
  "availability": "in_stock",
  "marketplace": "US",
  "source_url": "https://www.amazon.com/dp/B0EXAMPLE001",
  "fetched_at": "2026-08-17T09:30:00Z"
}

Treat the example above as a shape reference, not a guarantee. The exact fields depend on the worker, the marketplace region, and the current product page layout.

Pre-production checklist

Run this list before you connect the workflow to your CRM, queue, or AI agent:

[ ] Submit a one-source test job with three inputs and inspect each record manually.
[ ] Confirm every record originates from a publicly accessible URL.
[ ] Verify pricing, availability, and review fields match the public page on a sample of ten results.
[ ] Confirm the schema version you are using is documented and pinned.
[ ] Confirm your use case is consistent with the provider's terms and applicable privacy law.
[ ] Decide where results are stored, for how long, and who can access them.

Marketplace vs. Open-Source vs. Custom Build vs. Single-Source SaaS

Approach	Best for	Setup burden	Maintenance burden	Cost predictability
Ready-made marketplace	Small teams, mixed data sources, fast time-to-value	Low	Low: provider runs infrastructure	Medium: per-result or subscription
Open-source scraper you self-host	Engineering-heavy teams with one or two data sources	High	High: proxies, parsers, ops	Variable: depends on infra
Custom-built scraper	Unique data sources, no off-the-shelf worker exists	Highest	Highest: you own everything	Variable
Single-source SaaS API	One specific data source, no need for breadth	Low	Low: single integration	High: predictable

For teams that want a marketplace with pay-per-result pricing and ready-made workers, the CoreClaw product store lists the catalog and the CoreClaw pricing page describes current plans.

Business Use Cases

Lead generation. Run a marketplace job against a local-business source, pipe the records into your CRM, and route enriched leads to sales.
E-commerce monitoring. Track prices, stock, and review counts for a competitor catalog on a daily schedule.
SEO and rank tracking. Pull SERP results for a list of keywords and compare positions over time.
Market research. Aggregate public reviews, listings, or profiles into a research dataset for downstream analysis.
AI agent context. Provide public web data to an agent that drafts outreach, summaries, or comparisons from real inputs.

In every case the value is not the raw HTML, it is the structured, refreshable, reusable dataset.

Limitations and Compliance

A marketplace is not a workaround for access controls or private data. Keep these constraints in mind:

Public data only. Do not collect private messages, login-walled content, personal contact details not exposed publicly, or any data the source marks as private.
Terms and robots. Respect each source site's terms of service and robots directives. A marketplace operator should operate within the same boundaries.
Privacy law. Depending on jurisdiction, collecting and storing personal or business data may trigger GDPR, CCPA, or other privacy rules. Have a lawful basis and a retention policy.
Freshness. Public pages change. Marketplace results are a snapshot at the time of fetch, not a guarantee.
Regional coverage. Not every source has the same field set in every region.

For a broader view of public web data compliance and ethical scraping, see the CoreClaw public web data compliance guide (Chinese-language public-web-data reference).

FAQ

What is a scraper marketplace?
A hosted catalog of pre-built data extractors. Each entry targets one data source. You submit a job and receive structured JSON or CSV records back.

Is a marketplace the same as a single-source SaaS API?
No. A SaaS API usually covers one source. A marketplace covers many sources through one integration layer, with a consistent submission and delivery model.

How do I evaluate a marketplace without committing?
Run three small test jobs on sources you actually use. Compare the returned fields against what your application consumes, and check the schema is documented.

Can I deploy my own worker if no off-the-shelf one fits?
Many marketplaces let you write and deploy a custom worker on the same infrastructure. The CoreClaw worker creation flow lets you deploy a custom worker that runs in the same environment as the catalog.

What pricing model is best for a variable workload?
Pay-per-result usually wins for variable workloads because cost tracks delivered records. Confirm the rate on the provider's current pricing page or equivalent.

How do I keep results compliant with privacy law?
Collect only public data, store the minimum you need, set a retention limit, and document the lawful basis. Audit the pipeline annually as the law and the source sites change.

Can a marketplace feed an AI agent directly?
Yes. The structured JSON output is designed to be passed as context to an agent. Pin the schema version, document the data lineage, and include the source URL in each record for traceability.

Where to Go Next

If you are choosing a scraper marketplace for a real workload, the next step is to pick three sources you actually need, submit one test job against each, and compare the output against your downstream consumers.

Start with one source and one test job, validate the output, then expand the catalog. For a transparent view of plan options, see the CoreClaw pricing page.

LinkedIn Scraper API: How to Enrich B2B Leads Without Manual Research

coreclaw — Sat, 15 Aug 2026 01:04:04 +0000

LinkedIn Scraper API: How to Enrich B2B Leads Without Manual Research

A LinkedIn scraper API lets you turn public professional profiles and company pages into structured B2B lead data so your sales team can spend time selling instead of copy-pasting. If you are building outbound pipelines, recruiting workflows, or account-research tools, the right approach is not to run a browser automation script yourself; it is to send a structured request to a managed data API and receive normalized JSON you can push straight into a CRM, enrichment queue, or AI agent context.

This article shows what that workflow looks like, where the real operational limits are, and how to keep your data collection lawful and maintainable.

TL;DR

A LinkedIn scraper API returns structured public data — job titles, companies, locations, industries, company size, and public profile URLs — from professional pages.
The main alternative to a managed API is building headless-browser automation yourself, which means handling proxies, rate limits, session rotation, and layout changes.
A production workflow typically sends one request with search filters, polls for completion, then normalizes the JSON response for your CRM or enrichment tool.
Always collect only public data, respect LinkedIn's terms and robots directives, and comply with regional privacy rules.
For a ready-made B2B data pipeline, start with the CoreClaw product store and check the current CoreClaw pricing for pay-per-result options.

Why Manual LinkedIn Research Hits a Wall

Sales reps and founders know the drill: open a profile, read the headline, copy the name and title, switch tabs to find the company size, guess the industry, paste everything into a spreadsheet, and repeat. It is accurate in small doses, but it does not scale.

The real problems start when you need hundreds or thousands of prospects per week:

Fragmented data. A useful lead record combines profile fields, company attributes, and contact context that live on different pages.
Stale records. Roles change constantly. Manual exports become outdated before they are imported.
Operational drag. Every hour spent researching is an hour not spent on calls, demos, or follow-ups.
Infrastructure tax. If you try to automate this yourself, you immediately inherit proxy rotation, session management, CAPTCHA handling, and parser maintenance.

The search intent behind "LinkedIn scraper API" is not "how do I click faster." It is "how do I get reliable structured data without operating the scraping stack."

What Is a LinkedIn Scraper API?

A LinkedIn scraper API is a hosted service that accepts search parameters — such as job title, company, location, or industry — and returns structured records extracted from public LinkedIn pages. It is distinct from the official LinkedIn API, which requires partnership approval and is restricted to approved use cases. A scraper API focuses on publicly visible profile and company information that can be collected without authenticated access to private member data.

The output is usually JSON or JSONL and includes fields like:

profile_url — the public URL of the profile or company page
name — full name or company name
headline — current role or company tagline
current_company — employer or organization
location — geographic region
industry — sector classification
company_size — employee range, when visible
job_title — current position
description — public summary or about text

A managed API handles the rendering, parsing, and normalization for you. The trade-off is cost per result versus the engineering time and risk of running the infrastructure yourself.

A Production-Ready Python Workflow

The example below sends a request to a CoreClaw-compatible endpoint, polls for the result, and writes the structured records to a local JSON file. It uses environment variables for the endpoint and token so you never hardcode credentials or invent a URL.

import json
import os
import time

import requests

# Read configuration from the environment. Never commit real credentials.
API_TOKEN = os.environ["CORECLAW_API_TOKEN"]
ENDPOINT = os.environ["CORECLAW_LINKEDIN_ENDPOINT"]
POLL_INTERVAL = int(os.environ.get("CORECLAW_POLL_SECONDS", "10"))

# Search criteria: public professional profiles matching a role and region.
PAYLOAD = {
    "source": "linkedin",
    "filters": {
        "keywords": "sales director",
        "location": "United States",
        "industry": "Software Development",
    },
    "output_format": "json",
}


def submit_job(endpoint: str, token: str, payload: dict) -> str:
    response = requests.post(
        endpoint,
        headers={
            "Authorization": f"Bearer {token}",
            "Content-Type": "application/json",
        },
        json=payload,
        timeout=60,
    )
    response.raise_for_status()
    job_id = response.json().get("job_id")
    if not job_id:
        raise RuntimeError("No job_id returned by the endpoint.")
    return job_id


def fetch_results(endpoint: str, token: str, job_id: str) -> list[dict]:
    status_url = f"{endpoint.rstrip('/')}/{job_id}"
    while True:
        response = requests.get(
            status_url,
            headers={"Authorization": f"Bearer {token}"},
            timeout=60,
        )
        response.raise_for_status()
        data = response.json()
        status = data.get("status")
        if status == "completed":
            return data.get("results", [])
        if status in {"failed", "error"}:
            raise RuntimeError(f"Job failed: {data.get('message', 'unknown error')}")
        time.sleep(POLL_INTERVAL)


def save_records(records: list[dict], path: str) -> None:
    with open(path, "w", encoding="utf-8") as f:
        json.dump(records, f, indent=2, ensure_ascii=False)


def main() -> None:
    print("Submitting LinkedIn data job...")
    job_id = submit_job(ENDPOINT, API_TOKEN, PAYLOAD)
    print(f"Job submitted: {job_id}. Polling for completion...")

    records = fetch_results(ENDPOINT, API_TOKEN, job_id)
    print(f"Retrieved {len(records)} records.")

    save_records(records, "linkedin_leads.json")
    print("Saved to linkedin_leads.json")


if __name__ == "__main__":
    main()

Before running it, set your variables:

export CORECLAW_API_TOKEN="your_token_here"
export CORECLAW_LINKEDIN_ENDPOINT="https://console.coreclaw.com/api/v1/jobs"
export CORECLAW_POLL_SECONDS="10"
python linkedin_leads.py

The exact endpoint path depends on your CoreClaw account and the worker you are running. Copy the current endpoint from your CoreClaw console or worker settings rather than guessing a URL.

Representative Output

A single returned record might look like this:

{
  "profile_url": "https://www.linkedin.com/in/example-profile",
  "name": "Alex Morgan",
  "headline": "VP of Sales at SaaSFlow",
  "current_company": "SaaSFlow",
  "job_title": "VP of Sales",
  "location": "San Francisco, CA",
  "industry": "Software Development",
  "company_size": "51-200 employees",
  "public_identifier": "example-profile"
}

A complete job result is a list of such records. You can pipe the output directly into an enrichment tool, a CRM import, or an AI agent context.

Lead-record checklist

Before importing any records into your sales stack, verify each field:

[ ] The profile URL is publicly accessible without logging in.
[ ] The record contains only business-relevant attributes.
[ ] No private contact details, connection lists, or message content are included.
[ ] The data will be refreshed on a schedule that matches your sales cycle.
[ ] Your use case complies with LinkedIn's terms and applicable privacy law.

Build vs. Buy: Where the Cost Hides

Approach	Best for	Setup burden	Maintenance burden	Scaling risk
Manual research	One-off deals, tiny lists	None	Very high	Not scalable
Self-built scraper	Engineering teams with scraping expertise	Medium	High: proxies, parsers, rate limits	Moderate
Managed LinkedIn scraper API	Sales ops, founders, RevOps, AI-agent builders	Low	Low: provider handles infrastructure	Low

The managed API is usually the cheaper option once you account for engineering time, proxy spend, and the cost of rebuilding parsers after site changes. For teams that want a no-code starting point, the CoreClaw product store lists ready-made workers you can run without writing infrastructure code.

Business Use Cases

Outbound sales. Build account lists by industry, company size, and seniority, then pass structured records to your SDR tool.
Recruitment intelligence. Map talent pools by role, location, and employer without manually browsing profiles.
Account research. Enrich CRM company records with public employee counts, industries, and leadership profiles.
Partner discovery. Identify companies and decision-makers in a target ecosystem for channel or integration partnerships.
AI-agent context. Feed public business profiles into an agent that drafts personalized outreach or account summaries.

In every case, the value is not the raw HTML; it is the structured, refreshed, import-ready record.

Limitations and Compliance

A scraper API is not a workaround for access controls or private data. Keep the following constraints in mind:

Public data only. Do not attempt to collect private messages, connection graphs, email addresses hidden behind login walls, or any data marked as private by the platform.
Terms and robots. Respect LinkedIn's Terms of Service and robots directives. A managed provider should operate within the same boundaries.
Privacy law. Depending on your jurisdiction, collecting and storing professional data may trigger GDPR, CCPA, or other privacy rules. Have a lawful basis and a retention policy.
Rate and freshness. Public profile data changes daily. Results are a snapshot, not a guarantee of current employment.
Regional variation. Public page layouts and available fields differ by geography and account type, which can affect field coverage.

For a deeper look at public web data compliance, see the CoreClaw public web data compliance guide (Chinese-language public-web-data reference).

FAQ

Is there an official LinkedIn API for lead generation?
LinkedIn offers partner programs and restricted APIs, but general lead-generation access requires approval and is limited to approved use cases. A scraper API focuses on publicly visible data without authenticated member access.

What data fields are typically returned?
Public profile fields such as name, headline, current company, job title, location, industry, company size, and public profile URL. Private contact information and connection networks are out of scope.

How often should the workflow run?
For active pipelines, refresh target lists weekly or bi-weekly. For account research, monthly refreshes are usually enough. Match the cadence to your sales cycle and refresh only what you actually use.

Can I connect this to a CRM or n8n?
Yes. The JSON output is designed to be mapped into CRM imports, Google Sheets, Airtable, or automation platforms. Use the provider's documented integration or a simple webhook.

What happens when LinkedIn changes its layout?
A managed provider updates the parser. If you self-host, you own the fix. That difference is often the main reason teams switch to a hosted API.

How do I verify an endpoint before production use?
Start with a small test query, confirm the returned fields match your schema, and check that every record is from a public page. Only scale after validating output quality and legal compliance.

What should I look for in pricing?
Look for transparent per-result pricing rather than bundled proxy credits. Check the current CoreClaw pricing page to compare pay-per-result options.

Where to Go Next

If you are building B2B lead workflows, the next step is to choose a data source and a delivery target, then run a small test job before scaling.

Browse ready-made scrapers at the CoreClaw product store.
Compare pay-per-result plans on the CoreClaw pricing page.
Learn more about CoreClaw's approach to web data infrastructure at coreclaw.com.

Start with one role-and-region query, validate the output, and build your enrichment pipeline from there.

Google Maps Business Data in Python: How to Normalize Public Records for CRM Import

coreclaw — Thu, 13 Aug 2026 02:24:21 +0000

Google Maps Business Data in Python: How to Normalize Public Records for CRM Import

TL;DR: This article shows sales operations, RevOps, and Python developers how to take raw public business records from Google Maps and normalize them into a consistent JSON structure that is safe to import into a CRM. We use representative data, a runnable Python cleaning pipeline, and field-level mapping rules. The workflow is based on repositories maintained by scrapapi, not on a live production data service.

The Problem: Raw Public Records Are Messy

Public business listings are useful for lead research, but the JSON you collect is rarely CRM-ready. A single Google Maps record can carry the business name in name, title, or business_name; the phone in phone, phone_number, or telephone; and the address as a single string, a structured object, or a list of fragments.

If you import that data directly into a CRM, you end up with:

Duplicate companies created because names are spelled differently.
Missing phone numbers because the field name varies.
Addresses split incorrectly, breaking territory reports.
PII or private notes mixed into public fields.
Compliance questions because you cannot prove where a record came from.

This guide is for RevOps teams, sales operations, and Python developers who need a repeatable normalization step before any CRM import. The goal is not to fetch live data, but to transform representative public business records into a clean, auditable format.

What This Workflow Does

The scrapapi organization publishes example repositories that demonstrate how to work with public business data in Python. For this article we focus on three:

google-maps-shangjia-caiji — example patterns for handling public Google Maps business records.
python-google-maps-data — reusable Python helpers for parsing, normalizing, and validating business records.
crm-qiye-xinxi-buquan — field-mapping rules for enriching CRM company records with public business fields.

Together they show a workflow that:

Accepts raw JSON records from a public data source.
Normalizes names, phones, addresses, websites, and categories.
Maps public fields to CRM company fields without overwriting private data.
Adds provenance metadata so the record stays traceable.
Outputs a JSON or JSONL file ready for CRM import.

Important: These repositories contain example data-processing code. They are not production APIs and do not guarantee live-data coverage, rate limits, or platform compliance. Any production workflow requires a verified data source, authentication, observability, retries, and a data-governance review.

Setup

You only need Python 3.10 or newer. Create a virtual environment and a working directory:

python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install pydantic

The example below uses Pydantic for validation, but you can replace it with plain dictionaries if you prefer.

A Runnable Normalization Pipeline

The following Python script reads representative raw records, normalizes them, and writes CRM-ready JSON. It does not call any live API.

import json
import re
from typing import Any

from pydantic import BaseModel, Field, field_validator


class NormalizedBusiness(BaseModel):
    """A CRM-ready public business record."""

    source_id: str
    name: str = Field(..., min_length=1)
    phone: str | None = None
    website: str | None = None
    address: str | None = None
    city: str | None = None
    country: str | None = "US"
    categories: list[str] = Field(default_factory=list)
    raw_source: str = "google-maps-public-record"
    ingestion_date: str = "2026-08-13"

    @field_validator("phone")
    @classmethod
    def clean_phone(cls, value: str | None) -> str | None:
        if not value:
            return None
        digits = re.sub(r"\D", "", value)
        if len(digits) == 10:
            return f"+1-{digits[:3]}-{digits[3:6]}-{digits[6:]}"
        if len(digits) == 11 and digits.startswith("1"):
            return f"+1-{digits[1:4]}-{digits[4:7]}-{digits[7:]}"
        return value


def normalize_record(raw: dict[str, Any]) -> NormalizedBusiness:
    """Convert a raw public record into a normalized form."""

    # Accept multiple possible raw field names
    name = raw.get("name") or raw.get("title") or raw.get("business_name") or ""
    phone = raw.get("phone") or raw.get("phone_number") or raw.get("telephone")
    website = raw.get("website") or raw.get("website_url") or raw.get("url")

    # Address can be a string or a structured object
    address_value = raw.get("address") or raw.get("formatted_address") or ""
    if isinstance(address_value, dict):
        address_parts = [
            address_value.get("street"),
            address_value.get("city"),
            address_value.get("state"),
            address_value.get("postal_code"),
        ]
        address = ", ".join(part for part in address_parts if part)
        city = address_value.get("city")
    else:
        address = str(address_value)
        city = None

    # Categories can be a list or a single pipe-delimited string
    raw_categories = raw.get("categories") or raw.get("category") or []
    if isinstance(raw_categories, str):
        categories = [c.strip() for c in raw_categories.split("|") if c.strip()]
    else:
        categories = [str(c).strip() for c in raw_categories if c]

    return NormalizedBusiness(
        source_id=str(raw.get("place_id") or raw.get("id") or ""),
        name=name.strip(),
        phone=phone,
        website=website,
        address=address,
        city=city,
        categories=categories,
    )


def main():
    raw_records = [
        {
            "place_id": "gm_001",
            "name": "Downtown Coffee Roasters",
            "phone": "(555) 123-4567",
            "website": "https://downtowncoffee.example.com",
            "address": "1200 Market St, San Francisco, CA 94102",
            "categories": ["Coffee shop", "Cafe"],
        },
        {
            "id": "gm_002",
            "title": "Bayview Bakery",
            "phone_number": "5559876543",
            "url": "http://bayviewbakery.example.com",
            "formatted_address": {
                "street": "4500 3rd St",
                "city": "San Francisco",
                "state": "CA",
                "postal_code": "94158",
            },
            "category": "Bakery | Wholesale bakery",
        },
    ]

    normalized = [normalize_record(record).model_dump() for record in raw_records]

    with open("crm_ready_businesses.json", "w", encoding="utf-8") as f:
        json.dump(normalized, f, indent=2, ensure_ascii=False)

    print(json.dumps(normalized, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()

When you run the script, it produces two normalized records:

[
  {
    "source_id": "gm_001",
    "name": "Downtown Coffee Roasters",
    "phone": "+1-555-123-4567",
    "website": "https://downtowncoffee.example.com",
    "address": "1200 Market St, San Francisco, CA 94102",
    "city": null,
    "country": "US",
    "categories": ["Coffee shop", "Cafe"],
    "raw_source": "google-maps-public-record",
    "ingestion_date": "2026-08-13"
  },
  {
    "source_id": "gm_002",
    "name": "Bayview Bakery",
    "phone": "+1-555-987-6543",
    "website": "http://bayviewbakery.example.com",
    "address": "4500 3rd St, San Francisco, CA 94158",
    "city": "San Francisco",
    "country": "US",
    "categories": ["Bakery", "Wholesale bakery"],
    "raw_source": "google-maps-public-record",
    "ingestion_date": "2026-08-13"
  }
]

This output is representative. Your real input will have different field names, different address formats, and different category schemes. The important part is the normalization layer, not the sample data.

Field Mapping Without Overwriting Private CRM Data

A common mistake is to let public data overwrite fields that your sales team has already researched. Instead, map public fields into separate enrichment fields and let the CRM owner decide when to promote them.

Raw public field	Normalized field	CRM target field	Overwrite rule
`name`, `title`	`name`	`Company Name`	Only if CRM name is blank
`phone`, `phone_number`	`phone`	`Public Phone`	Add if missing; do not replace verified phone
`website`, `url`	`website`	`Company Website`	Only if blank or flagged as outdated
`address`, `formatted_address`	`address`	`Public Address`	Add as secondary address
`categories`, `category`	`categories`	`Industry Tags`	Merge, deduplicate

The crm-qiye-xinxi-buquan repository explores this mapping in more detail, including rules for handling Chinese-language company records and privacy-sensitive fields.

Use Cases

Local lead generation: Normalize a list of businesses in a target neighborhood before loading them into HubSpot, Salesforce, or Pipedrive.
Territory planning: Extract city, state, and postal code from address strings so the CRM can assign records to the right sales rep.
Deduplication prep: Standardize phone numbers and websites so matching algorithms can identify duplicate companies.
AI agent context: Provide a clean JSON record that an LLM-based agent can reason about without being exposed to raw, inconsistent source data.

Compliance and Limitations Checklist

Before importing any public data into a CRM, verify the following:

[ ] The data comes from a public source and does not include private or scraped-from-private-source records.
[ ] You have reviewed the target platform's terms of service and robots policy.
[ ] You respect applicable privacy laws for the regions where the businesses and your CRM users are located.
[ ] You keep provenance metadata so every record is traceable to its source and ingestion date.
[ ] You do not claim coverage, accuracy, or freshness that you cannot verify.
[ ] You separate public enrichment fields from privately researched fields to avoid overwriting sales intelligence.
[ ] You have retries, observability, and rate-limit handling if your production pipeline calls an external service.

This article and the referenced repositories focus only on public web data and public business information. They do not cover hacking, account cracking, private data, stolen databases, CAPTCHA bypass, or evading access controls.

Frequently Asked Questions

Q1: Does this code fetch live Google Maps data?

No. The Python script above processes representative JSON records stored in the file. It does not call any live API or unverified endpoint. A production workflow needs a verified data source and proper authentication.

Q2: Can I import the output directly into HubSpot or Salesforce?

The JSON structure is designed to be import-friendly, but you should still map the fields to your CRM's import format. Always test with a small batch first and validate phone numbers and addresses.

Q3: How do I handle records with missing phones or addresses?

Keep the record and mark the missing fields as null. Do not invent data. You can add a data_quality_score field to flag records that need manual review.

Q4: Should I overwrite existing CRM company names with public data?

Generally no. Public data should land in enrichment fields. Let the sales or RevOps team decide when to promote a public value to the primary field.

Q5: What if the raw record uses a language other than English?

The normalization logic is language-agnostic for structure, but you may need locale-specific rules for phone numbers, address order, and category translation. The crm-qiye-xinxi-buquan repository includes examples for Chinese-language company records.

Q6: How do I prove where a record came from during an audit?

Store source_id, raw_source, and ingestion_date in the normalized record and, if possible, in the CRM itself. Provenance metadata is the simplest way to make public data auditable.

Q7: Where can I find more example workflows?

Browse the scrapapi GitHub profile. The python-google-maps-data repository has additional parsing utilities, and google-maps-shangjia-caiji shows example patterns for handling Google Maps business records.

Start Normalizing Public Business Records

If you are building a RevOps or sales-operations workflow around public business data, the repositories maintained by scrapapi give you a starting point:

google-maps-shangjia-caiji for example Google Maps record patterns.
python-google-maps-data for reusable Python normalization helpers.
crm-qiye-xinxi-buquan for CRM field-mapping and enrichment rules.

Remember to treat these as example data-processing tools, not production data sources. For production, add authentication, observability, retries, rate-limit handling, and a data-governance review.

Zillow Scraper API: How to Build a Real Estate Property Monitor

coreclaw — Thu, 13 Aug 2026 02:05:57 +0000

Zillow Scraper API: How to Build a Real Estate Property Monitor

If you need to track property listings, price changes, or listing status over time, the most practical starting point is a Zillow scraper API workflow that converts public listing pages into structured, time-stamped records. This article is for developers, data analysts, and real estate operations teams who want to build a property monitor from repository-based Python examples rather than maintaining a full headless-browser stack themselves.

Quick answer

A real estate property monitor does three things:

Collect structured fields from public Zillow listing pages on a schedule you define.
Store each observation with a collection timestamp and source URL so you can detect changes.
Compare new records against previous ones to surface price drops, status changes, and new listings.

The data-scrape GitHub organization hosts two relevant reference repositories for this workflow: the Zillow Scraper API project and the companion Zillow Data Scraper. Treat these as implementation references, not guaranteed production services. Verify the README, code, and current maintenance status before relying on any endpoint or field.

Why public real estate data is harder to monitor than it looks

Zillow's public pages are designed for human browsing, not programmatic feeds. Anyone building a monitor quickly runs into the same problems:

Dynamic rendering. Listing details, photos, and pricing history are loaded by JavaScript after the initial page request. A plain HTTP client often receives incomplete HTML.
Layout drift. CSS selectors and page structure change without warning, which breaks extraction logic within weeks or months.
Rate limiting and session behavior. Repeated requests from the same IP or user-agent pattern can trigger blocks, empty responses, or verification challenges.
Data freshness. A listing may already be under contract, removed, or re-listed by the time your monitor runs. The page timestamp and your collection timestamp are different facts.
Duplicate and variant records. The same address can appear as a new listing, a price change, or a re-post, so a monitor needs de-duplication rules.

These constraints make self-hosted scraping viable for prototypes but expensive to maintain at scale. A repository-based scraper API workflow gives you a middle ground: you run the code, control the schema, and decide how much infrastructure to own.

What a Zillow scraper API returns

A "scraper API" in this context is a wrapper around the extraction layer. You send a property URL, search parameters, or location query; the API returns normalized JSON. The exact fields depend on the repository implementation and the page type, but typical listing-level output includes:

Property identity

zillow_id or listing identifier
address, city, state, zip_code
latitude, longitude when available
property_url (the canonical source URL)

Listing details

price, price_history (when exposed on the page)
bedrooms, bathrooms, square_footage
listing_status: for sale, pending, sold, for rent, etc.
days_on_market or listing date when visible

Media and source

photo_urls (first image or gallery list)
description text from the public listing
agent_name or brokerage when publicly displayed

Provenance

collected_at: the time your workflow observed the page
source: the public page origin
review_status: a label for human validation

Because public pages differ by region, device type, and whether the listing is active, treat any field as optional until you have validated it against a controlled sample.

Step-by-step: building the monitor

1. Set environment variables

export ZILLOW_API_ENDPOINT="https://your-endpoint-or-localhost.example.com/scrape"
export ZILLOW_API_KEY="your_api_key_if_required"
export TARGET_ZIP="90210"
export MONITOR_OUTPUT_DIR="./data"

Replace the endpoint with the current value shown in the repository README or your own deployment.

2. Request listing data

import json
import os
import hashlib
from datetime import datetime, timezone
from pathlib import Path

import requests

API_ENDPOINT = os.environ["ZILLOW_API_ENDPOINT"]
API_KEY = os.environ.get("ZILLOW_API_KEY")
OUTPUT_DIR = Path(os.environ.get("MONITOR_OUTPUT_DIR", "./data"))
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

headers = {"Content-Type": "application/json"}
if API_KEY:
    headers["Authorization"] = f"Bearer {API_KEY}"

# Example payload; confirm the exact schema in the repository README.
payload = {
    "query": "for-sale",
    "location": os.environ.get("TARGET_ZIP", "90210"),
    "max_results": 25,
}

response = requests.post(API_ENDPOINT, headers=headers, json=payload, timeout=120)
response.raise_for_status()
raw_listings = response.json()
print(f"Retrieved {len(raw_listings.get('data', []))} listings")

3. Normalize and store records

def clean_text(value):
    if not isinstance(value, str):
        return None
    value = " ".join(value.split())
    return value or None


def as_positive_int(value):
    if isinstance(value, bool):
        return None
    if isinstance(value, int) and value > 0:
        return value
    if isinstance(value, str):
        digits = "".join(c for c in value if c.isdigit())
        if digits:
            return int(digits)
    return None


def normalize_listing(raw, collected_at):
    address = clean_text(raw.get("address") or raw.get("streetAddress"))
    if not address:
        return None

    price = as_positive_int(raw.get("price") or raw.get("unformattedPrice"))

    record = {
        "listing_id": clean_text(raw.get("zpid") or raw.get("listing_id") or raw.get("id")),
        "address": address,
        "city": clean_text(raw.get("city")),
        "state": clean_text(raw.get("state")),
        "zip_code": clean_text(raw.get("zipcode") or raw.get("zip")),
        "price": price,
        "bedrooms": as_positive_int(raw.get("bedrooms") or raw.get("beds")),
        "bathrooms": as_positive_int(raw.get("bathrooms") or raw.get("baths")),
        "square_feet": as_positive_int(raw.get("livingArea") or raw.get("square_feet")),
        "status": clean_text(raw.get("status") or raw.get("listingStatus")),
        "property_url": clean_text(raw.get("detailUrl") or raw.get("property_url") or raw.get("url")),
        "photo_url": clean_text(raw.get("imgSrc") or raw.get("photo_url")),
        "collected_at": collected_at,
        "source": "public Zillow listing page",
        "review_status": "needs_review",
    }
    return record


collected_at = datetime.now(timezone.utc).isoformat()
records = []
seen_ids = set()

for raw in raw_listings.get("data", []):
    if not isinstance(raw, dict):
        continue
    record = normalize_listing(raw, collected_at)
    if record and record["listing_id"] and record["listing_id"] not in seen_ids:
        records.append(record)
        seen_ids.add(record["listing_id"])

output_file = OUTPUT_DIR / f"listings_{datetime.now(timezone.utc).strftime('%Y%m%d_%H%M%S')}.json"
output_file.write_text(json.dumps(records, indent=2, ensure_ascii=False), encoding="utf-8")
print(f"Saved {len(records)} normalized records to {output_file}")

4. Detect changes across runs

def record_fingerprint(record):
    """Create a stable key for de-duplication and change detection."""
    parts = [
        record.get("listing_id") or "",
        str(record.get("price") or ""),
        record.get("status") or "",
    ]
    return hashlib.sha256("|".join(parts).encode()).hexdigest()[:16]


for record in records:
    record["fingerprint"] = record_fingerprint(record)

# A production monitor would compare fingerprints against the previous run
# and emit events for new listings, price changes, and status changes.

This separation keeps extraction, normalization, and change detection independent. If the extraction layer changes, you can update one function without rewriting the monitor logic.

Representative output shape

After normalization, a single record looks like this:

{
  "listing_id": "123456789",
  "address": "1234 Sunset Blvd",
  "city": "Los Angeles",
  "state": "CA",
  "zip_code": "90210",
  "price": 2495000,
  "bedrooms": 4,
  "bathrooms": 3,
  "square_feet": 3100,
  "status": "FOR_SALE",
  "property_url": "https://www.zillow.com/homedetails/1234-sunset-blvd-los-angeles-ca-90210/123456789_zpid/",
  "photo_url": "https://photos.zillowstatic.com/.../image.jpg",
  "collected_at": "2026-08-13T02:00:00+00:00",
  "source": "public Zillow listing page",
  "review_status": "needs_review",
  "fingerprint": "a1b2c3d4e5f67890"
}

A null field means the value was not present or not parseable in that run, not that it is zero or empty. Keeping that distinction visible prevents bad downstream decisions.

Use cases for a property monitor

Price-drop alerts. Compare daily fingerprints and notify a buyer or investor when a listing price decreases.

Market velocity tracking. Count new listings, pending sales, and sold listings per ZIP code over time to estimate inventory turnover.

Rental arbitrage research. Monitor rental listings, price per square foot, and days on market for specific neighborhoods.

Investment screening. Filter by bedroom count, square footage, and status to build a shortlist for human review.

Comparable sales analysis. Collect recently sold records with price and property attributes to support valuation models.

Each use case should start with a narrow scope, clear data retention rules, and a manual review gate before any financial or legal decision.

Repository workflow versus building from scratch

Dimension	Repository-based scraper API	Build your own from scratch
Setup time	Hours to days, depending on the repo	Days to weeks for a reliable prototype
Maintenance	You update selectors and dependencies	You own the full extraction and anti-detection stack
Schema control	You define the normalization layer	You define everything
Proxy/headless overhead	Depends on repo design; may still need proxies	You manage browsers, proxies, and rotation
Field certainty	Validate from actual test outputs	Validate from your own parsing logic
Compliance	Your responsibility either way	Your responsibility either way

Neither option removes the obligation to respect Zillow's terms, applicable law, and privacy requirements. A repository gives you a head start on code; it does not give you permission to collect or use data indiscriminately.

Limitations and compliance

No official Zillow API guarantee. Public pages are not a stable data contract. Fields, URLs, and availability can change.
Regional variation. The same query may return different fields or layouts depending on location, device fingerprint, and session.
Rate and volume limits. Running a monitor too aggressively can trigger blocks. Start with a small cadence and increase only after observing stable behavior.
Data accuracy. Public listings can be stale, duplicated, or re-listed. Always validate high-impact records before acting.
Legal and platform terms. Use public data only for lawful purposes and in compliance with Zillow's terms of use, robots directives, and applicable privacy laws. Do not evade access controls, logins, or rate limits.

FAQ

Is the repository a managed production API?

No. The Zillow Scraper API repository and Zillow Data Scraper repository are code references. Review their READMEs, dependencies, and recent commits before using them in any system.

What fields are guaranteed?

None. Public page extraction depends on the current page structure, region, and listing type. Validate every field you plan to use with a controlled test sample.

How often should the monitor run?

Start daily. Increase frequency only if the decision you are supporting actually benefits from hourly updates and you can handle the increased block risk.

Can this feed into a CRM, dashboard, or AI agent?

What happens when Zillow changes its page layout?

The extraction step may break or return partial records. A well-structured monitor detects increased null rates or parse failures and alerts a human to inspect the repository for updates.

Should I keep every raw response?

Only retain what your use case and governance rules justify. Keep enough to audit a result, but do not accumulate unnecessary personal or property data.

Where do I inspect the referenced projects?

Start with the data-scrape GitHub profile, then review the Zillow Scraper API repository and the Zillow Data Scraper repository.

Next step

Before running a production monitor, define the exact ZIP codes or search criteria you need, write a target JSON schema, collect a small test sample, and assign someone to validate the output. The repositories above are useful starting points for that evaluation; the durable asset is the documented, reviewable workflow your team builds around them.

TikTok Data API: How to Extract Trends, Creators, and Engagement

coreclaw — Thu, 13 Aug 2026 01:04:21 +0000

TikTok Data API: How to Extract Trends, Creators, and Engagement

If you need structured TikTok data — trending hashtags, creator profiles, video engagement metrics — the most reliable path for production teams is a managed TikTok data API that returns normalized JSON without requiring you to maintain headless browsers, proxy pools, or anti-detection logic. This article is written for developers, data analysts, and growth teams who want to feed TikTok signals into dashboards, CRMs, or AI agent workflows.

Quick Answer

Use a managed web data API that handles proxy rotation, browser fingerprinting, and JSON normalization for you. You send a search or profile request; the API returns structured fields such as view count, like count, follower count, hashtags, and post timestamps. For teams building trend pipelines or creator research tools, this removes the operational burden of maintaining a scraper against a platform that changes its markup and anti-bot behavior frequently.

Why Collecting TikTok Data Is Harder Than It Looks

TikTok's public pages load large amounts of JavaScript and enforce aggressive rate limiting. If you try to fetch a profile or hashtag page with a simple HTTP client, you typically get:

Empty or obfuscated HTML — the data is rendered client-side after multiple JS requests.
Rate limits and IP blocks — repeated requests from the same IP trigger temporary blocks or CAPTCHA challenges.
Layout drift — class names and DOM structures change without warning, breaking CSS selectors within weeks.
Session and cookie requirements — some pages expect specific headers or cookies that are hard to replicate programmatically.

These issues multiply when you scale from one-off research to a daily trend-monitoring pipeline. Self-hosted solutions require headless browser infrastructure, proxy rotation, and a maintenance team that can react to platform changes within hours.

What a TikTok Data API Returns

A managed TikTok data API abstracts the scraping layer and returns normalized JSON. The exact fields depend on the endpoint — profile, video, hashtag, or trend — but typical responses include:

Profile-level fields

Username, display name, bio, avatar URL
Follower count, following count, total likes received
Verified status and profile URL

Video-level fields

Video URL, description, hashtags, and music/sound metadata
View count, like count, comment count, share count, save count
Post timestamp, duration, and resolution

Hashtag and trend-level fields

Hashtag name, total video count, total view count
Related hashtags and top videos under the tag
Growth velocity signals when the API supports time-series data

Because the API normalizes the response, you can store the data in a database, stream it to a dashboard, or feed it into an AI agent without writing custom parsers for each platform update.

Step-by-Step: Extracting TikTok Data with Python

The workflow below assumes you have access to a managed web data API. You supply an endpoint URL and API key from your provider's console, then send search or profile requests.

1. Set Up Environment Variables

Never hardcode credentials. Store the endpoint and API key in environment variables:

export CORECLAW_API_KEY="your_api_key"
export CORECLAW_ENDPOINT="https://your-provider.example.com/v1/tiktok"

Replace the endpoint with the current value from your provider's console or product documentation.

2. Request a Creator Profile

import os
import requests
import json

API_KEY = os.environ.get("CORECLAW_API_KEY")
ENDPOINT = os.environ.get("CORECLAW_ENDPOINT", "https://api.example.com/v1/tiktok")

headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json",
}

# Request a profile by username
payload = {
    "query": "username",
    "value": "techcreator",
    "type": "profile",
}

response = requests.post(ENDPOINT, headers=headers, json=payload, timeout=60)
response.raise_for_status()

profile = response.json()
print(json.dumps(profile, indent=2, ensure_ascii=False))

3. Request Hashtag Trend Data

# Request videos under a hashtag, sorted by engagement
payload = {
    "query": "hashtag",
    "value": "productivity",
    "type": "videos",
    "max_results": 50,
    "sort_by": "likes",
}

response = requests.post(ENDPOINT, headers=headers, json=payload, timeout=120)
response.raise_for_status()

videos = response.json()
print(f"Retrieved {len(videos.get('data', []))} videos")

4. Build a Simple Trend Monitor

from datetime import datetime, timedelta

def track_hashtag_growth(hashtag: str, days: int = 7):
    """Compare top video engagement over a rolling window."""
    payload = {
        "query": "hashtag",
        "value": hashtag,
        "type": "videos",
        "max_results": 100,
        "sort_by": "date",
    }
    resp = requests.post(ENDPOINT, headers=headers, json=payload, timeout=120)
    resp.raise_for_status()

    videos = resp.json().get("data", [])
    cutoff = datetime.utcnow() - timedelta(days=days)

    recent = [v for v in videos if datetime.fromisoformat(v["create_time"].replace("Z", "+00:00")) > cutoff]
    total_views = sum(v.get("view_count", 0) for v in recent)
    total_likes = sum(v.get("like_count", 0) for v in recent)
    avg_engagement = (total_likes / total_views * 100) if total_views else 0

    return {
        "hashtag": hashtag,
        "videos_count": len(recent),
        "total_views": total_views,
        "total_likes": total_likes,
        "avg_engagement_rate": round(avg_engagement, 2),
    }

# Example usage
report = track_hashtag_growth("aitools", days=7)
print(json.dumps(report, indent=2))

Expected Output Schema

Below is a representative JSON shape returned by a managed TikTok data API. Actual fields vary by provider and endpoint; confirm the current schema in your provider's documentation.

{
  "data": [
    {
      "id": "7421234567890123456",
      "url": "https://www.tiktok.com/@techcreator/video/7421234567890123456",
      "description": "5 AI tools that save me 10 hours a week #aitools #productivity",
      "create_time": "2026-08-08T14:30:00Z",
      "duration": 45,
      "view_count": 850000,
      "like_count": 62000,
      "comment_count": 3400,
      "share_count": 8900,
      "save_count": 12000,
      "hashtags": ["aitools", "productivity", "ai"],
      "music": {
        "title": "original sound - techcreator",
        "artist": "techcreator"
      },
      "author": {
        "username": "techcreator",
        "display_name": "Tech Creator",
        "followers": 450000,
        "following": 120,
        "verified": true,
        "bio": "AI and automation tips every week"
      }
    }
  ],
  "pagination": {
    "has_more": true,
    "next_cursor": "eyJwYWdlIjoyfQ=="
  }
}

Business Use Cases

Trend Detection and Content Strategy

Marketing teams monitor hashtags and trending sounds to surface content angles before they peak. A daily API call for top hashtags in a niche — combined with engagement velocity — gives a data-driven editorial calendar.

Creator Discovery and Influencer Research

Agencies build creator databases by querying profiles by keyword, follower range, and engagement rate. Structured data makes it easy to filter and rank candidates without manual profile browsing.

Competitor Monitoring

Track competitor brand mentions, branded hashtag campaigns, and top-performing video formats. Knowing what resonates for rival accounts reduces trial-and-error in your own content.

Market Research and Product Intelligence

Consumer brands watch TikTok for organic product reviews, unboxing trends, and usage patterns that surface faster than on traditional review platforms.

AI Agent and Automation Workflows

Feed TikTok trend data into LLM-based agents or no-code automation tools such as n8n. Structured JSON is easier to parse into prompts, CRM records, or Slack alerts than raw HTML.

Build vs. Buy: Comparison Checklist

Dimension	Self-Built Scraper	Managed TikTok Data API
Setup time	Days to weeks (browser, proxy, parser)	Minutes (copy endpoint, set key)
Maintenance	Continuous (layout changes, anti-bot updates)	Provider handles anti-detection
Proxy and IP management	Source and rotate residential or mobile proxies	Included in the API layer
Output format	Raw HTML or partial JSON you normalize	Normalized JSON with stable schema
Scale	Limited by your proxy budget and concurrency	Provider concurrency and caching
Compliance risk	You manage robots.txt and terms interpretation	Provider enforces public-data boundaries
Cost predictability	Variable (proxy bills, dev hours, infra)	Pay-per-result or subscription model

For one-off scripts, a self-built approach can work. For production pipelines, research teams, or SaaS products, a managed API is usually cheaper when you factor in maintenance time.

Limitations, Freshness, and Compliance

What You Should Know Before Production

Public data only — A reputable TikTok data API collects only publicly visible metadata. It does not access private accounts, direct messages, or personal information behind login walls. You must respect TikTok's Terms of Service and applicable privacy laws such as GDPR and CCPA.
No official API guarantee — TikTok does not offer a general-purpose public API for all video and profile data. The official Research API has a lengthy application process, strict use-case restrictions, and limited regional availability. A managed data API is a practical alternative, not a replacement for an official partnership.
Freshness and caching — Responses may be cached for minutes to hours. If you need real-time counts for a live campaign, verify the provider's cache window before committing to a schedule.
Regional coverage — Trending content varies by country. Some APIs allow geo-targeting; others return global aggregates. Confirm whether the dataset supports the regions your team cares about.
Layout changes — Even managed APIs can experience temporary gaps when a platform rolls out a major UI update. Reputable providers mitigate this, but no scraper is immune to structural change.
Rate limits and quotas — Every API enforces concurrency and volume limits. Understand your expected request volume before building a daily pipeline.

FAQ

Is there an official TikTok API for this data?

TikTok offers a Research API for approved academic and commercial research use cases, but the application process is slow and the data scope is limited. Most production teams use a managed web data API for broader, faster access.

What data fields are returned?

Typical fields include username, bio, follower count, video URL, description, hashtags, view count, like count, comment count, share count, music metadata, and post timestamp. Verify the exact schema with your provider before production use.

How often should I run the workflow?

For trend monitoring, daily polling is common. For creator research, weekly or on-demand calls are usually sufficient. Match your schedule to the data freshness your use case requires.

What happens when TikTok changes its page layout?

Managed API providers handle anti-detection and parser updates as part of their service. Self-built scrapers break immediately and require manual fixes.

Can this connect to Python, n8n, or an AI agent?

Yes. The JSON output is designed for programmatic consumption. You can pipe it into a Python pandas workflow, an n8n automation, or an LLM agent as structured context.

What should I verify before production use?

Confirm the endpoint's regional coverage, cache window, rate limits, and the provider's compliance stance on public data. Test with a small dataset before scaling your pipeline.

Do I need to manage proxies or headless browsers?

No. A managed data API handles proxy rotation, browser fingerprinting, and JSON normalization. You send requests and receive structured data.

Ready to Extract TikTok Data Without the Infrastructure Overhead?

If your team needs structured TikTok data for trend analysis, creator research, or AI agent workflows, a managed web data platform saves you from building and maintaining fragile scrapers.

Browse the CoreClaw ready-made workers store to find TikTok and other social data scrapers you can run immediately.
See CoreClaw pricing for pay-per-result options that scale with your volume.
Visit coreclaw.com to learn how production teams use CoreClaw as their web data infrastructure for AI agents and automation.

YouTube Video Metadata API: A Python Workflow for Structured Creator Research

coreclaw — Wed, 12 Aug 2026 02:07:56 +0000

YouTube Video Metadata API: A Python Workflow for Structured Creator Research

A useful creator-research workflow starts with a stable record format, not a spreadsheet of copied links. If your team needs to review public YouTube videos across a defined set of channels or topics, model every result as structured metadata: the source URL, title, publication time when available, observed metrics, collection timestamp, and a clear provenance note. That gives researchers a dataset they can validate, enrich, and re-run without losing track of where each value came from.

This guide is for developers and research teams who want to turn public video results into a reviewable JSON workflow. It uses the repository examples in the data-scrape GitHub organization, especially the YouTube Video Scraper API project and the related YouTube Channel Scraper. The repositories provide implementation references; they are not a guarantee of a production data service, complete field coverage, or a fixed endpoint contract.

Quick answer

For structured creator research:

Define the video fields you need before collecting anything.
Store every public result with its source URL and collection time.
Normalize missing values rather than inventing defaults.
Keep extraction, transformation, and analyst review as separate steps.
Use a repository-based scraper only after validating its README, code, current maintenance status, and target-platform requirements.

The key outcome is a dataset that can answer a specific question, such as: Which topics appeared in a competitor channel's latest public videos? Which videos should an analyst review manually? Which records are too incomplete to use?

Why a JSON workflow beats ad-hoc exports

A CSV is fine for a one-off review. It becomes fragile when records arrive over several runs, values change, or different people need to understand how a number was collected. JSON keeps related fields together and makes provenance explicit.

A minimal public-video record can look like this:

{
  "video_url": "https://www.youtube.com/watch?v=example",
  "channel_url": "https://www.youtube.com/@example",
  "title": "Example public video title",
  "published_at": null,
  "observed_views": null,
  "collected_at": "2026-08-12T02:00:00Z",
  "source": "public YouTube page",
  "review_status": "needs_review"
}

This shape makes uncertainty visible. null means the field was unavailable or not yet verified; it does not mean zero. collected_at separates the time you observed a value from the video publication date. review_status keeps the data pipeline from quietly turning raw web output into a business conclusion.

Start with an explicit research question

Do not begin by collecting every field that a tool might expose. Start with a question and only keep data needed to answer it.

Research question	Minimum useful fields	Human validation needed?
Find recent videos about a product category	title, video URL, channel URL, observed publication time	Yes, for topical relevance
Create a manual creator-review queue	title, channel name, video URL, collection time	Yes, before outreach or decisions
Compare publishing cadence	channel URL, public video URL, observed publication time, collection time	Yes, for edge cases and deleted content
Build a content taxonomy	title, description when public, URL, reviewer label	Yes, for category accuracy

This constraint is useful for engineering too. It reduces work, limits retention of unnecessary data, and makes a failed or partial extraction easier to diagnose.

Repository roles: channel discovery and video records

The YouTube Channel Scraper repository documents a Python-based project for extracting publicly available channel information and video URLs. Its README shows Python and command-line examples, and describes JSON, CSV, and Excel output options.

The YouTube Video Scraper API repository documents a separate project focused on video-data extraction, with JSON responses, pagination, Docker deployment, Swagger documentation, rate limiting, and batch-video lookup listed in its README. Verify the source code and current documentation before adopting any request format, deployment behavior, or response field in production: a repository README is an implementation reference, not a service-level agreement.

A sensible separation is:

Channel stage: produce a candidate list of public channel or video URLs.
Video stage: collect and normalize video-level records for those URLs.
Research stage: classify and review the records using explicit rules.

That separation lets you replace one component when page structures, dependencies, or target-site requirements change.

Install and inspect before integrating

The channel repository documents installation from GitHub for Python 3.8+:

pip install git+https://github.com/data-scrape/youtube-channel-scraper.git

The video project documents a similar repository installation pattern:

pip install git+https://github.com/data-scrape/youtube-video-scraper-api.git

Before wiring either into an automated job, review four things:

The latest README and dependency list.
Open issues and recent commits for maintenance signals.
The exact returned fields in a controlled test run.
Your legal, privacy, and target-platform obligations for the intended collection.

Avoid treating an example import, a repository name, or an illustrative scrape("your-query") call as proof that every query format, field, or output is supported in your environment.

Normalize raw results with a small Python transformer

The following code does not call a scraper or claim any live-data endpoint. It demonstrates the safer middle stage: transforming representative public-video records into a consistent JSON schema that your team can review.

import json
import os
from datetime import datetime, timezone
from pathlib import Path

INPUT_FILE = Path(os.environ.get("VIDEO_INPUT", "raw_videos.json"))
OUTPUT_FILE = Path(os.environ.get("VIDEO_OUTPUT", "normalized_videos.json"))


def clean_text(value):
    if not isinstance(value, str):
        return None
    value = " ".join(value.split())
    return value or None


def as_non_negative_int(value):
    if isinstance(value, bool):
        return None
    if isinstance(value, int) and value >= 0:
        return value
    if isinstance(value, str) and value.strip().isdigit():
        return int(value.strip())
    return None


def normalize_video(raw, collected_at):
    url = clean_text(raw.get("video_url") or raw.get("url"))
    if not url:
        return None

    return {
        "video_url": url,
        "channel_url": clean_text(raw.get("channel_url")),
        "channel_name": clean_text(raw.get("channel_name")),
        "title": clean_text(raw.get("title")),
        "description": clean_text(raw.get("description")),
        "published_at": clean_text(raw.get("published_at")),
        "observed_views": as_non_negative_int(raw.get("view_count")),
        "observed_likes": as_non_negative_int(raw.get("like_count")),
        "collected_at": collected_at,
        "source": "public web result",
        "review_status": "needs_review",
    }


raw_records = json.loads(INPUT_FILE.read_text(encoding="utf-8"))
if not isinstance(raw_records, list):
    raise ValueError("Expected a JSON array of raw video records")

collected_at = datetime.now(timezone.utc).isoformat()
normalized = []
seen_urls = set()

for raw in raw_records:
    if not isinstance(raw, dict):
        continue
    record = normalize_video(raw, collected_at)
    if record and record["video_url"] not in seen_urls:
        normalized.append(record)
        seen_urls.add(record["video_url"])

OUTPUT_FILE.write_text(json.dumps(normalized, indent=2, ensure_ascii=False), encoding="utf-8")
print(f"Saved {len(normalized)} normalized records to {OUTPUT_FILE}")

The transformer makes three practical decisions: it removes whitespace-only values, retains unknown fields as null, and removes exact duplicate URLs. It does not assume that a missing metric means a zero metric, or that a date string has already been validated.

Representative input and output

A raw record from any collection stage may use inconsistent field names or types:

[
  {
    "url": "https://www.youtube.com/watch?v=example",
    "channel_name": "Example Creator",
    "title": "  Public video title  ",
    "view_count": "1200",
    "like_count": 80
  }
]

After normalization, the record becomes easier to work with downstream:

[
  {
    "video_url": "https://www.youtube.com/watch?v=example",
    "channel_url": null,
    "channel_name": "Example Creator",
    "title": "Public video title",
    "description": null,
    "published_at": null,
    "observed_views": 1200,
    "observed_likes": 80,
    "collected_at": "2026-08-12T02:00:00+00:00",
    "source": "public web result",
    "review_status": "needs_review"
  }
]

At this point, a reviewer can validate relevance, mark invalid records, or add a taxonomy label. That is far safer than letting a raw title or estimated engagement number drive a sales, editorial, or marketing action automatically.

Build reliability checks into the workflow

Public web pages change. Fields may disappear, videos may be removed, pages may render differently by region, and a repository may need maintenance after a dependency update. Treat these as routine conditions, not exceptions.

Use a pre-run checklist:

[ ] Confirm that the repository and its dependencies are still maintained enough for a test run.
[ ] Test against a small, approved set of public URLs.
[ ] Record collection time and source URL for each result.
[ ] Keep missing fields as missing rather than substituting assumptions.
[ ] Deduplicate by canonical video URL where possible.
[ ] Add retries and error logs only after you know the tool's documented behavior.
[ ] Have a human review high-impact records before publishing, outreach, or business decisions.

For recurring work, measure record completeness and error rates over time. A sudden jump in missing titles or URLs is often more meaningful than the raw number of collected rows.

Common use cases

Content research. Build a manual queue of public videos around a topic, then label them by format, audience, or product mention.

Creator-marketplace operations. Collect a reviewable set of public channel and video URLs, then ask a human operator to validate fit before contact or partnership decisions.

Media analysis. Preserve source URLs and collection times so a report can distinguish observed values from conclusions.

Developer prototyping. Use repository examples to test a JSON schema and transformation process before committing to a production data source.

None of these use cases should be read as permission to collect restricted information, ignore platform controls, or automate actions against people without a lawful basis.

Repository workflow versus a managed data provider

Dimension	Repository-based workflow	Managed data provider
Control over schema	You define transformation and storage	Provider schema guides integration
Maintenance	Your team monitors code and page changes	Provider typically manages its service layer
Observability	You design logs, tests, and alerts	Varies by provider plan and product
Field certainty	Validate from actual test outputs	Validate from current provider documentation
Compliance review	Your responsibility	Still your responsibility; provider tooling does not replace it

A repository can be an excellent learning or prototyping asset. Production adoption still needs authentication handling where required, error monitoring, data retention rules, access controls, and a documented basis for using each type of public information.

Limitations and compliance

Use only publicly available information in a way that respects applicable law, privacy requirements, and the target platform's terms. Do not try to evade login requirements, technical restrictions, rate limits, or access controls. Do not use public-profile information to infer sensitive traits or make consequential automated decisions.

Repository examples can change, and public page data is not automatically complete or correct. Verify the latest project documentation and test output before relying on a field, a supported format, an authentication pattern, or a claimed capability.

FAQ

Is a GitHub repository a production API?

No. A repository may contain examples, code, or deployment instructions. Review its code and documentation before deciding whether it fits a production system.

Why save `collected_at` when video pages have their own dates?

A page publication date and the time your workflow observed a value are different facts. Keeping both helps analysts understand freshness and changes over time.

Can I automatically treat missing views as zero?

No. A missing value may mean the field was unavailable, the parser changed, the source was incomplete, or the record needs manual review.

Should I retain every raw response?

Only retain what your research purpose, governance rules, and legal obligations justify. Keep enough provenance to audit a result without accumulating unnecessary data.

How often should a recurring collection run?

Choose a cadence based on the actual decision being supported, platform requirements, and the cost of reviewing changes. Test a small cadence first and adjust based on observed value.

Where can I inspect the referenced projects?

Start with the data-scrape profile, then review the YouTube Video Scraper API repository and YouTube Channel Scraper repository directly.

Next step

Before integrating a public-video workflow, write the target JSON schema, collect a controlled test sample, and assign someone to validate output quality. The repository examples above are useful starting points for that evaluation; the durable asset is the documented, reviewable workflow your team builds around them.

YouTube Channel Scraper: How to Build Creator Analytics with Python

coreclaw — Wed, 12 Aug 2026 01:03:02 +0000

A YouTube channel scraper is the practical starting point when a creator research, partnerships, or competitive analysis team needs repeatable channel-level data — uploads, cadence, durations, visible engagement signals — without spending the data team on endless manual checks. The reliable approach is to define the channel-level fields you actually need, decide whether the YouTube Data API v3 is enough or whether you outgrow it, collect observations on a schedule that matches the decision, and store each collection with a timestamp. This guide is for developers, growth analysts, and creator-economy teams who want a Python pipeline that turns one channel URL into a comparable analytics record.

TL;DR

A channel-scoped workflow answers questions that single-video scrapers cannot. You want to know, per channel: how often new public videos appear, how long those videos are, which categories and topics recur, and how visible engagement shifts over time. The official YouTube Data API v3 is the cheapest path for small, stable channel sets, but its unit quota evaporates once you start tracking thousands of videos or pulling comment replies. A managed YouTube channel scraper is a reasonable option when the team prefers to outsource browser execution, parsing, and infrastructure operations. Keep credentials and any provider endpoint outside source code, and always verify the current product behavior on the official console and pricing pages before production use.

Why Channel-Level Data Is Harder Than It Looks

Many teams start by scraping individual video pages and assume the channel view falls out for free. It does not. A few examples:

A single video shows you one data point. A channel shows you a publishing pattern — what kinds of videos a creator publishes, on what cadence, with what range of durations, and with what visible engagement responses.
Engagement ratios are only meaningful across a channel. A video with 50,000 views tells you nothing on its own; compared with the creator's median views, it tells you whether the format outperformed or underperformed.
Manual review cannot keep up with channels that upload weekly. Without an automated schedule, the team quietly starts reading fewer and fewer new videos, and the dataset drifts toward the most recent ones only.
Channel-level fields like description, banner, links, and country are often stored on a different page (/about) or a different response shape than video-level fields, which means your scraper needs to handle at least two distinct schemas.

The net effect is that a "watch this creator in a browser tab" workflow stops scaling the moment the team has more than a handful of channels. A channel-scoped pipeline is what makes the work sustainable.

What Channel Data Means

Before choosing a tool, separate the object you want to collect from the decision you want to enable.

Object	Typical public fields	Decision it supports
Channel header	display name, handle, description, country, join date, banner, vanity URL, verified status	Creator inventory and account classification
Channel video list	per video: ID, title, published timestamp, duration, view count, like count, category	Publishing cadence, format mix, performance drift
Per-video comments	text, author handle, like count, reply count, published timestamp	Audience sentiment, recurring questions, moderation review
Channel about	links, email, contact, country, view count, subscriber count	Outreach targets, partnership qualification

The field names above are a planning model, not a guarantee that every provider or page returns every value. Visible counts, subscriber figures, and country metadata can change between page renders, and not every channel exposes every section. Define required fields, acceptable fallbacks, and a retention policy before production use.

Step 1: Anchor the Workflow to a Research Question

A broad request such as "track YouTube creators" is not operational. Replace it with questions that can be answered from public channel data:

Which public tech channels with similar audience size moved from long-form to Shorts during the last quarter, and how did view counts change?
What is the median upload cadence and median video duration of the top ten creators in a niche, and how is that pattern shifting?
Which channels are repeatedly publishing comparison posts about a specific product category, and what does the visible engagement say about audience interest?

For each question, write down the seed list of channels, the target fields, the cadence, and the owner. A useful request record may look like this:

{
  "channels": ["@example_creator_a", "@example_creator_b"],
  "include": ["channel_header", "recent_videos", "video_stats"],
  "max_videos_per_channel": 30,
  "collected_at": "2026-08-12T01:00:00Z",
  "purpose": "niche creator benchmarking"
}

Do not add private channels, login tokens for any account, or sensitive personal data to this workflow. Limit collection to public channel data that serves a documented business purpose.

Step 2: Keep Provider Settings Outside Your Code

The following Python example is intentionally provider-neutral. It demonstrates the integration boundary, the per-channel loop, pagination, validation, and JSON Lines storage. It does not assume a specific CoreClaw endpoint name, SDK, response format, quota, or account configuration.

Set the endpoint shown in your approved provider console or documentation, then export it in your environment:

export CORECLAW_API_KEY="your-key"
export CORECLAW_YOUTUBE_CHANNEL_ENDPOINT="https://your-approved-endpoint"

On Windows PowerShell, use $env:CORECLAW_API_KEY and $env:CORECLAW_YOUTUBE_CHANNEL_ENDPOINT for the current session. Keep these values out of Git and out of any public example.

Step 3: Collect Public Channel and Video Records

import json
import os
from datetime import datetime, timezone
from pathlib import Path

import requests

API_KEY = os.environ["CORECLAW_API_KEY"]
ENDPOINT = os.environ["CORECLAW_YOUTUBE_CHANNEL_ENDPOINT"]
OUTPUT = Path("youtube_channel_public_data.jsonl")

HEADERS = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json",
}

REQUEST = {
    "channels": ["@example_creator_a", "@example_creator_b"],
    "include": ["channel_header", "recent_videos", "video_stats"],
    "max_videos_per_channel": 30,
}


def normalize_channel(channel: dict, collected_at: str) -> dict:
    """Keep only public fields your approved response actually supplies."""
    return {
        "object_type": "channel_header",
        "channel_handle": channel.get("handle"),
        "display_name": channel.get("title"),
        "description": channel.get("description"),
        "country": channel.get("country"),
        "subscriber_count": channel.get("subscriber_count"),
        "view_count": channel.get("view_count"),
        "joined_at": channel.get("joined_at"),
        "is_verified": channel.get("is_verified"),
        "collected_at": collected_at,
    }


def normalize_video(video: dict, collected_at: str) -> dict:
    return {
        "object_type": "video",
        "channel_handle": video.get("channel_handle"),
        "video_id": video.get("video_id"),
        "title": video.get("title"),
        "published_at": video.get("published_at"),
        "duration_seconds": video.get("duration_seconds"),
        "view_count": video.get("view_count"),
        "like_count": video.get("like_count"),
        "category": video.get("category"),
        "collected_at": collected_at,
    }


def fetch_page(payload: dict) -> dict:
    response = requests.post(ENDPOINT, headers=HEADERS, json=payload, timeout=45)
    response.raise_for_status()
    return response.json()


def main() -> None:
    collected_at = datetime.now(timezone.utc).isoformat()
    cursor = None
    written = 0

    with OUTPUT.open("a", encoding="utf-8") as destination:
        while True:
            payload = dict(REQUEST)
            if cursor:
                payload["cursor"] = cursor

            page = fetch_page(payload)
            items = page.get("items", [])

            for item in items:
                if item.get("object_type") == "channel_header":
                    destination.write(json.dumps(normalize_channel(item, collected_at)) + "\n")
                    written += 1
                elif item.get("object_type") == "video":
                    destination.write(json.dumps(normalize_video(item, collected_at)) + "\n")
                    written += 1

            cursor = page.get("next_cursor")
            if not cursor or not items:
                break

    print(f"Wrote {written} public channel/video records to {OUTPUT}")


if __name__ == "__main__":
    main()

The only provider-specific code should be the request payload and the normalizer. Some providers return an array under data, others under items; some expose a cursor, others a page number. Make those differences explicit in a tested adapter instead of hiding them behind assumptions. For example, sample response shapes for planning:

{
  "items": [
    {
      "object_type": "channel_header",
      "handle": "@example_creator_a",
      "title": "Example Creator A",
      "description": "Public channel description",
      "country": "US",
      "subscriber_count": 482000,
      "view_count": 71320000,
      "joined_at": "2018-04-02",
      "is_verified": true
    },
    {
      "object_type": "video",
      "channel_handle": "@example_creator_a",
      "video_id": "abc123xyz",
      "title": "How a creator analytics pipeline actually works",
      "published_at": "2026-08-04T13:00:00Z",
      "duration_seconds": 932,
      "view_count": 412000,
      "like_count": 14800,
      "category": "Education"
    }
  ],
  "next_cursor": null
}

Treat this as a planning shape. Confirm the actual fields against your provider's documented response before adapting the normalizer.

Step 4: Derive Channel Metrics From Observations

Raw rows are useful for storage. The analytics team needs derived channel-level metrics:

Median views per video over the last 30 days, plus the 10th and 90th percentile.
Upload cadence, computed as the median number of days between consecutive uploads.
Median duration and the share of videos under 60 seconds, a useful proxy for Shorts share.
Engagement ratio, computed as median(likes) / median(views) per channel, with the caveat that visible like counts can be turned off by the creator.

Compute these on top of the stored rows, not inside the scraper. Keeping the scraper focused on collection makes it easier to swap providers later without rewriting analytics code.

Step 5: Choose a Cadence That Matches the Decision

A weekly review of ten channels is fine for a monthly partnership report. A daily review of one hundred channels is reasonable for an early-warning trend dashboard. Tighter cadences are not automatically better. They increase the chance that the team acts on normal short-term variation in visible engagement counts.

Use the slowest cadence that supports the decision. Record the time zone used for windows. Do not compare a video collected minutes after publication with another collected days later as if the values were equivalent.

Build Yourself or Use a Managed Channel Scraper?

Three approaches are realistic for a channel-scoped workflow in 2026.

Approach	Best for	What your team maintains	Main tradeoff
YouTube Data API v3 only	A small, stable channel set with low volume	Quota budgeting, request shaping, response parsing	Default unit quota runs out fast; comment threads and some metadata are limited
Self-hosted scraping with `yt-dlp` or hidden endpoints	A unique workflow with engineering capacity	Browser runs, parsers, layout-change handling, proxy management	Maximum control, ongoing maintenance, anti-detection work
Managed channel scraper API	Repeatable, multi-channel collection with minimal ops	Input validation, storage, analytics, compliance review	Confirm data shape and current commercial terms

If you prefer a ready-made worker rather than maintaining browser execution in-house, browse the CoreClaw Workers Store for reusable collection workers. When you want to deploy a custom worker on CoreClaw's own infrastructure rather than run one yourself, the CoreClaw console is the starting point for a new worker.

For current product availability and pricing details, verify them directly on the official pricing page before committing a workflow. Pricing, limits, and product behavior can change; this article intentionally does not claim fixed values.

Compliance and Data Quality Boundaries

Public visibility does not remove responsibility. Before launching a production workflow, review the target platform's terms, applicable privacy law, your contractual obligations, and the intended use of each field. Do not use collected channel data to infer sensitive traits, automate high-impact decisions, or contact people in ways that conflict with applicable rules.

Useful safeguards:

Collect only fields that serve a documented purpose.
Limit access to raw comment text, especially if comments contain personal data.
Set a retention period and delete channel records that are no longer needed.
Keep a human reviewer responsible for any partnership or outreach conclusion.
Monitor failures and schema changes so missing fields are not mistaken for real behavioral changes.
Be careful with subscriber and view counts. Both are visible on the channel page but can be inflated or rounded by the platform.

FAQ

Is the YouTube Data API v3 enough for channel analytics?

For a small, stable channel set and modest volume, yes. For higher volume, deeper comment data, or layout-driven enrichment, the quota and field limits push teams toward an additional channel-scoped scraper. Review your own usage profile and the platform's documented limits before deciding.

What data should I collect for creator research?

Start with the channel header fields, recent video metadata, and the visible engagement counts. Avoid collecting more than you need, especially anything tied to private community posts, member-only content, or per-user analytics.

How often should the workflow run?

Pick the slowest cadence that still supports the decision. Weekly cadence is often enough for partnership reviews; daily cadence is justified only when the team actively acts on short-term changes.

What happens when layouts or response fields change?

Use a schema validation check on the response, keep representative approved responses under test where permitted, and alert when required keys disappear or types change. Your analytics code should treat optional fields as optional.

Can this connect to a CRM, dashboard, or AI workflow?

Yes, once normalized. Pass approved rows to your warehouse, dashboard, CRM, or an AI-assisted review queue. Keep the connection scoped to public, relevant channel data and retain human oversight over any business action.

How is a channel scraper different from a video scraper?

A video scraper focuses on one video page at a time. A channel scraper aggregates across the channel's header, recent video list, and per-video metrics so that cadence, format mix, and engagement ratios can be computed. Most creator analytics use cases need the channel view, not just the video view.

How do I evaluate a managed YouTube channel scraper?

Confirm supported channel objects, expected response shape, pagination, error handling, current pricing, permitted use, and data freshness expectations. Run a small pilot with representative channels before wiring it into a recurring workflow.

Next Step

Start with a short channel list and a narrow research question. Validate the returned schema, establish a retention policy, then scale only when the data improves a real business decision. Review the YouTube channel scraper for the channel-level fields and cadence metrics described above, browse the Workers Store for adjacent use cases, and use the CoreClaw console to deploy a custom worker when your workflow outgrows a single script.

Instagram Scraper API: How to Extract Posts, Profiles, and Comments

coreclaw — Tue, 11 Aug 2026 02:12:30 +0000

An Instagram scraper API is useful when you need repeatable, structured public data from profiles, posts, and comments for creator research, brand monitoring, or campaign analysis. The practical approach is to define the public fields you need, collect them through a provider you have approved, paginate carefully, and store each collection with a timestamp. This guide is for developers and marketing-data teams who want a Python workflow without turning one-off browser checks into an unreliable manual process.

TL;DR

Profile, post, and comment data solve different questions. Profiles help you discover and classify creators or public brand accounts. Posts help you measure topics, formats, publishing cadence, and visible engagement signals. Comments add qualitative context, but they also require more careful use because content can include personal information.

A managed service can remove the recurring work of browser execution, response parsing, and infrastructure operations. For a product-oriented starting point, evaluate the CoreClaw Instagram scraper alongside your own requirements for fields, freshness, geography, and permitted use. Keep credentials and any account-specific endpoint configuration outside source code.

Why an API Workflow Is Better Than Manual Checking

Manual research has a place when you are reviewing a few posts. It becomes weak when a team needs to answer the same question every week:

Which creators regularly discuss a product category?
Which themes appear in public brand posts this month?
How did the visible engagement pattern change after a campaign?
Which posts need a human review because comments contain repeated questions or complaints?

A browser tab does not give you a consistent data model, history, or reproducible query. An API workflow can. It lets you collect a documented request, normalize the result into a small schema, keep an audit trail, and feed analysis into a dashboard, CRM, spreadsheet, or internal database.

That does not mean the data is complete or permanent. Public pages change, fields can disappear, and providers can return different shapes. Treat each run as an observation at a point in time, not as a permanent truth about a person or business.

Profile, Post, and Comment Data Are Different Datasets

Before choosing a tool, separate the object you want to collect from the decision you want to make.

Dataset	Useful public fields	Typical decision
Profile	handle, display name, biography, external link, visible follower count, category	Creator discovery or brand-account inventory
Post	URL, caption, media type, published time, visible likes or comments, hashtags	Content analysis and campaign monitoring
Comment	text, timestamp, public author handle, reply relationship	Question themes, audience feedback, moderation review

The field names above are a planning model, not a promise that every provider or page will return every value. For example, visible counts, pagination behavior, and media metadata can vary by page type and the access method you configure. Define required fields, acceptable fallbacks, and a record-retention policy before production use.

Step 1: Define a Small, Specific Research Question

Start with a question that can be answered from public data. A broad request such as "track Instagram" is not operational. These are better:

Find public skincare creators who posted about a specific ingredient during the last campaign period.
Track the publishing cadence and visible engagement of ten competitor brand accounts.
Collect public comments on selected launch posts, then route repeated support questions to a human reviewer.

For each question, specify the seed input, target output, cadence, and owner. A useful request record may look like this:

{
  "seed_profiles": ["example_brand", "example_creator"],
  "include": ["profiles", "recent_posts"],
  "max_pages": 3,
  "collected_at": "2026-08-11T01:10:00Z",
  "purpose": "public campaign research"
}

Do not add private account data, login credentials for target accounts, or sensitive personal data to this workflow. Limit collection to public information that is relevant to a defined business purpose.

Step 2: Keep Provider Settings Outside Your Code

The following Python example is intentionally provider-neutral. It demonstrates the integration boundary, pagination loop, validation, and JSON Lines storage. It does not assume a CoreClaw endpoint name, SDK, response format, quota, or account configuration.

Set the endpoint shown in your approved provider console or documentation, then export it in your environment:

export CORECLAW_API_KEY="your-key"
export CORECLAW_INSTAGRAM_ENDPOINT="https://your-approved-endpoint"

On Windows PowerShell, use $env:CORECLAW_API_KEY and $env:CORECLAW_INSTAGRAM_ENDPOINT for the current session. Keep these values out of Git and out of any public example.

Step 3: Collect Public Objects With Pagination

import json
import os
from datetime import datetime, timezone
from pathlib import Path

import requests

API_KEY = os.environ["CORECLAW_API_KEY"]
ENDPOINT = os.environ["CORECLAW_INSTAGRAM_ENDPOINT"]
OUTPUT = Path("instagram_public_data.jsonl")

HEADERS = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json",
}

REQUEST = {
    "profiles": ["example_brand", "example_creator"],
    "include": ["profile", "posts", "comments"],
    "max_pages": 3,
}


def normalize_record(item: dict, collected_at: str) -> dict:
    """Keep only fields that your approved response actually supplies."""
    return {
        "object_type": item.get("object_type"),
        "source_url": item.get("url"),
        "handle": item.get("handle"),
        "text": item.get("caption") or item.get("text"),
        "published_at": item.get("published_at"),
        "visible_like_count": item.get("like_count"),
        "visible_comment_count": item.get("comment_count"),
        "collected_at": collected_at,
    }


def fetch_page(cursor: str | None = None) -> dict:
    payload = dict(REQUEST)
    if cursor:
        payload["cursor"] = cursor

    response = requests.post(ENDPOINT, headers=HEADERS, json=payload, timeout=45)
    response.raise_for_status()
    return response.json()


def main() -> None:
    collected_at = datetime.now(timezone.utc).isoformat()
    cursor = None
    written = 0

    with OUTPUT.open("a", encoding="utf-8") as destination:
        while True:
            page = fetch_page(cursor)
            items = page.get("items", [])

            for item in items:
                destination.write(json.dumps(normalize_record(item, collected_at)) + "\n")
                written += 1

            cursor = page.get("next_cursor")
            if not cursor or not items:
                break

    print(f"Wrote {written} public records to {OUTPUT}")


if __name__ == "__main__":
    main()

The only provider-specific code should be the request payload and the normalizer. Inspect a real approved response before deciding which keys to keep. Some providers return an array under data, others under items; some expose a cursor, others a page number. Make those differences explicit in a tested adapter instead of hiding them behind assumptions.

Step 4: Store Observations, Not Just Snapshots

A useful dataset needs two time dimensions: when the post or comment was published, and when your system collected it. The first supports content chronology. The second helps you understand freshness and changes in visible counts.

For ongoing monitoring, store at least:

a stable source URL or provider object ID when available;
the public handle and object type;
published_at if supplied;
collected_at from your own job;
the visible engagement values returned in that run;
the request configuration that produced the row.

Appending JSON Lines is a reasonable first step. For larger workflows, move the same normalized schema into SQLite, Postgres, BigQuery, or your analytics warehouse. Do not overwrite previous measurements if trend analysis is the goal.

Step 5: Choose a Collection Cadence That Matches the Decision

A creator-discovery dataset may only need a periodic refresh. A launch-monitoring dashboard may need a shorter interval during an active campaign. More frequent calls are not automatically better: they increase cost, volume, and the chance that your team acts on normal short-term variation.

Use the slowest cadence that supports the decision. Record the time zone used for campaign windows, and avoid comparing a post collected minutes after publication with another collected days later as if the values were equivalent.

Build Yourself or Use a Managed Scraper?

There are two different jobs here. Your team owns the data model, the business question, storage, and compliance review. The collection provider owns its documented integration and the infrastructure it offers. A CoreClaw Workers Store can be worth evaluating when you prefer selecting a ready-made worker rather than maintaining browser automation and response handling in-house.

Approach	Best for	What your team maintains	Main tradeoff
Manual review	A few accounts and one-off research	Notes and process consistency	Does not scale or create history well
Self-hosted automation	A unique workflow with engineering capacity	Browser runs, parsers, failures, and changes	Maximum control, ongoing maintenance
Managed scraper API	Repeatable structured collection	Input validation, storage, analysis, and policy review	Confirm data shape and current commercial terms

For current product availability and pricing details, verify them directly on the CoreClaw pricing page before committing a workflow. Pricing, limits, and product behavior can change; this article intentionally does not claim fixed values.

Compliance and Data Quality Boundaries

Public visibility does not remove responsibility. Before launching a production workflow, review the target platform's terms, applicable privacy law, your contractual obligations, and the intended use of each field. Do not use collected content to infer sensitive traits, make automated high-impact decisions, or contact people in ways that conflict with applicable rules.

Use sensible safeguards:

Collect only fields that serve a documented purpose.
Limit access to raw comments, especially if they contain personal details.
Set a retention period and delete records that are no longer needed.
Give a human reviewer responsibility for sentiment labels, campaign conclusions, and outreach decisions.
Monitor failures and schema changes so missing fields are not mistaken for real behavioral changes.

FAQ

Is there an official Instagram API for every public profile and post?

Platform-provided APIs and access models have their own eligibility, permissions, and field rules. A managed scraper API is a separate integration model. Review the documentation and policies for the approach you choose rather than assuming one provides the same coverage as the other.

What data should I collect for creator research?

Start with public profile identifiers, post URLs, publishing times, visible engagement values, and topical text that supports the research question. Avoid collecting more than you need, and keep the collection purpose documented.

How should I handle comments?

Treat comments as qualitative input. Sample or classify them with a human review process, restrict raw access, and do not turn public remarks into unsupported conclusions about individuals.

What happens when page layouts or response fields change?

Use a schema validation check, keep representative approved responses in tests where permitted, and alert when required keys vanish or types change. Your downstream analysis should tolerate optional fields.

Can this connect to a CRM or AI workflow?

Yes, once you have a normalized schema, you can pass approved rows to your warehouse, CRM, or an AI-assisted review queue. Keep the connection scoped to public, relevant data and retain human oversight over any business action.

How do I evaluate a managed Instagram scraper?

Confirm the supported public objects, expected response shape, pagination, error handling, current pricing, permitted use, and data freshness expectations. Run a small pilot with representative accounts before wiring it into a recurring workflow.

Next Step

Start with a small set of public profiles and a narrow research question. Validate the returned schema, establish a retention policy, then scale only when the data improves a real business decision. Review the CoreClaw Instagram scraper, browse the Workers Store, and verify current terms on the pricing page before production use.

Google Maps Scraper GitHub: How to Extract Business Data with Python

coreclaw — Tue, 11 Aug 2026 02:05:39 +0000

Google Maps Scraper GitHub: How to Extract Business Data with Python

If you are searching for a Google Maps scraper GitHub repository that extracts business listings with Python, the short answer is: yes, the open-source scrape-google-maps repository provides a starting point for collecting business names, phones, websites, addresses, ratings, reviews, opening hours, and coordinates from publicly visible Maps results. This article is for developers, lead generation agencies, local SEO specialists, and data engineers who want to control the scraping pipeline themselves rather than pay per record to a managed API.

You will see exactly what the repository covers, which fields you can extract, how to wire up a Python environment to run it, what output to expect, the tradeoffs versus commercial Google Maps APIs, and where to extend the workflow with related repositories in the same organization.

Quick Answer (TL;DR)

The scrape-google-maps repository on GitHub is an open-source Python toolkit for pulling business listings from Google Maps search results. Supply a query like "plumbers in Austin, TX," and receive structured records with name, phone, website, address, rating, review count, hours, and geographic coordinates. Pair it with the yellow-pages-scraper to cross-check a market against a directory source. Browse the data-scrape GitHub organization for the full inventory.

Why Google Maps Data Is Hard to Pull Consistently

Google Maps is the largest business directory on the public web. It is also one of the hardest sources to scrape reliably. Three structural challenges explain the difficulty:

Pagination is not based on a simple ?page= parameter. Results are split across "search this area" actions, infinite-scroll feeds, and cluster expansions. A naive script that requests one URL once will see only the first 20-40 results.
Field visibility is gated by interactions. Ratings, review counts, phone numbers, and opening hours live inside expandable panels. Click events must be simulated to expose the full record.
Anti-bot defenses evolve continuously. Google fingerprint browsers, IP ranges, request headers, and interaction timing. A scraper that runs today may fail tomorrow without a config update.

This is exactly why an open-source, Pythonic implementation with documented configuration is more valuable than a one-off script. You can read the source, follow the change log, and adapt to upstream changes.

What the scrape-google-maps Repository Covers

The scrape-google-maps repository automates Google Maps search-result extraction in Python. Based on its README, it can collect business name, phone, website, category, rating, review count, address, opening hours, latitude and longitude, place ID (where exposed), and image URLs. The repository introduces both self-hosted scraping approaches and pointers to production-ready APIs for engineers who outgrow a self-hosted run. Verify the exact field set in the latest README — it can change with each release.

If your goal also involves directory data, the companion yellow-pages-scraper covers business name, phone, address, category, ratings, reviews, social links, and hours of operation. The two together form a useful cross-source validation pair when deduplicating or enriching Maps-only leads.

Step-by-Step Setup

Prerequisites

Python 3.10 or higher
A virtual environment tool (venv, uv, or conda)
A network path to Google Maps (some regions and IP ranges get blocked more aggressively than others; a residential proxy may be required at scale)
Familiarity with environment variable configuration

Install

git clone https://github.com/data-scrape/scrape-google-maps.git
cd scrape-google-maps
python -m venv .venv
source .venv/bin/activate     # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Always read the requirements.txt file shipped with the repository to confirm the exact dependency versions. The snippet above is generic; the actual file may pin specific package versions.

Environment Configuration

Create a .env file in the project root:

QUERY=Plumbers in Austin, TX
MAX_RECORDS=200
OUTPUT_PATH=./output/austin_plumbers.json
PROXY_URL=                     # Optional: leave empty for first run
USER_AGENT=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
LOG_LEVEL=INFO

Never hard-code credentials, proxies, or production search terms in source. Use environment variables and rotate them through your deployment or CI pipeline.

Runnable Python Example

The following is a generic, runnable client for a hypothetical Scraper class shipped by the repository. Adapt the import path and configuration knobs to match the version you cloned. The example demonstrates the standard pattern: load config from the environment, run a search, and write structured output to disk.

import json
import os
from pathlib import Path

# Replace with the actual entry point exported by scrape-google-maps
# in the version you have cloned. The import path below is illustrative.
from scraper import Scraper  # type: ignore

QUERY = os.environ.get("QUERY", "Plumbers in Austin, TX")
MAX_RECORDS = int(os.environ.get("MAX_RECORDS", "200"))
OUTPUT_PATH = Path(os.environ.get("OUTPUT_PATH", "./output/results.json"))
PROXY_URL = os.environ.get("PROXY_URL") or None

def load_config():
    return {
        "query": QUERY,
        "max_records": MAX_RECORDS,
        "proxy": PROXY_URL,
        "lang": "en",
        "region": "US",
        "headless": True,
    }

def normalize(record: dict) -> dict:
    """Trim whitespace and drop None values for cleaner downstream storage."""
    return {k: (v.strip() if isinstance(v, str) else v) for k, v in record.items() if v not in (None, "", [])}

def main():
    config = load_config()
    OUTPUT_PATH.parent.mkdir(parents=True, exist_ok=True)

    scraper = Scraper(proxy=config["proxy"], headless=config["headless"])
    records = []
    try:
        for record in scraper.search(
            query=config["query"],
            max_records=config["max_records"],
            lang=config["lang"],
            region=config["region"],
        ):
            records.append(normalize(record))
            print(f"Collected: {record.get('name', '<unknown>')}")
    finally:
        scraper.close()

    with OUTPUT_PATH.open("w", encoding="utf-8") as f:
        json.dump(records, f, indent=2, ensure_ascii=False)
    print(f"Wrote {len(records)} records to {OUTPUT_PATH}")

if __name__ == "__main__":
    main()

What This Script Does

Reads every knob from the environment, including the search query, output path, and optional proxy.
Initializes the scraper with a proxy if one is configured; omit it for low-volume first runs.
Streams results from the scraper as they arrive, normalizes whitespace and empties, and prints a progress line per record.
Writes the collected records to JSON, which is the simplest format for piping into pandas, DuckDB, or a CRM webhook.
Closes the scraper in a finally block so browser resources are released even on error.

If the actual repository exposes a different API surface (for example, an async client or CLI wrapper), prefer the README's quick-start command first, then graduate to a typed configuration like the one above.

Expected Output Fields

A normalized output record typically looks like this (illustrative example; verify against the current schema in the repository):

{
  "name": "Example Plumbing Co.",
  "phone": "+1 512-555-0142",
  "website": "https://example-plumbing.com",
  "category": "Plumber",
  "rating": 4.7,
  "review_count": 213,
  "address": "123 Main St, Austin, TX 78701, USA",
  "opening_hours": ["Mon-Fri 08:00-18:00", "Sat 09:00-14:00"],
  "latitude": 30.2711,
  "longitude": -97.7437,
  "place_id": "ChIJ_example_placeholder",
  "source_query": "Plumbers in Austin, TX"
}

Treat this as a structural reference, not a contract. Field names and types can change with the implementation. Always inspect the first few records you collect and write a schema validation step before trusting the output.

Business Use Cases

Google Maps data underpins several commercial workflows:

Local lead generation. Build targeted B2B lists by industry and geography for outbound sales teams. Cross-reference with the yellow-pages-scraper to catch businesses that are present in directories but absent from Maps.
Local SEO and citation audits. Compare NAP (Name, Address, Phone) consistency across directories, identify duplicate listings, and surface inconsistent categories.
Market sizing. Count businesses by category and city to estimate addressable market size before launching a new service in a region.
Competitive mapping. Identify chains, franchises, and independents in a territory; collect ratings and review counts to benchmark service quality.
Enrichment for AI agents. Feed structured business records into RAG pipelines that answer location-aware questions ("which HVAC companies are open right now in zip 78701?").

Build vs Buy: Open-Source Scraper vs. Managed API

Neither path is universally better. Use the same dimensions to compare for any Google Maps data project:

Dimension	Open-Source Scraper (this repo)	Managed Google Maps API
Best for	Teams with Python skills who want full control and tolerate maintenance	Teams that need SLA-backed uptime and minimal DevOps
Setup model	Clone the repo, install Python deps, supply proxy and config	Subscribe, obtain an API key, send REST requests
Data coverage	Whatever the open-source scraper currently collects from public result pages	Whatever the API endpoint exposes; verify against current docs
Output format	JSON/CSV you control; pipe to any storage	JSON over HTTP, structured to the API's schema
Maintenance burden	You monitor upstream changes and update selectors when pages change	Provider handles upstream changes
Quota considerations	Limited by your infrastructure and proxy capacity	Per-request quota; verify current numbers with the provider
Pricing verification	Verify VPS and proxy costs with your providers	Confirm current pricing on the provider's official pricing page

The open-source route is often cheaper at scale. A managed API is faster to integrate. Many teams prototype with the open-source scraper, then graduate to an API once the workflow is validated.

Limitations, Compliance, and Maintenance

Running a Google Maps scraper in production is a sustained engineering investment:

Terms of service and legal posture. Review Google's terms and the laws in your jurisdiction before running any scraper at scale. Publicly visible data collected for internal analysis is generally treated differently from data sold or republished. This article is not legal advice; consult counsel for your specific use.
Selector drift. Google Maps markup changes periodically. Any field may disappear or rename without notice. Build a validation layer that checks for the presence of each expected field and alerts when collection rates drop.
Rate limiting and pagination. Aggressive request patterns can trigger throttling or IP blocks. Add jitter between requests, cap concurrency, and respect any site-specific limits you observe.
Phone and email extraction. Publicly listed phone numbers are usually present in Maps results. Emails are rarer and typically require a downstream lookup against the business's own website.
Geographic coverage. Some countries and languages have weaker data coverage, different field availability, and stricter anti-bot defenses. Test against your target region before committing to a long-running pipeline.
Storage and PII. Business contact data may be subject to data-protection rules depending on jurisdiction. Document a retention policy and provide an opt-out path.
Anti-bot detection. Headless browsers are detectable. The --disable-blink-features=AutomationControlled flag and a realistic user agent reduce — but do not eliminate — flagging. Residential proxies and human-paced interaction are essential for high-volume targets.

FAQ

Is there an official Google Maps scraper GitHub project maintained by Google?

No. Google publishes the Places API for programmatic access; it does not maintain an open-source Google Maps scraper. The scrape-google-maps repository is a community-built Python project.

What data fields does it extract?

Typical fields include business name, phone, website, category, rating, review count, address, opening hours, latitude, longitude, and place ID. Confirm the exact list against the current README and validate with a sample run.

Can this scale to 100,000 records?

Scale is a function of infrastructure, proxy quality, and pacing discipline. Most teams start with batches of a few thousand records, observe how Google responds, and then increase cadence. Plan for queue-based concurrency control rather than parallel-for-loop blasting.

How is this different from the yellow-pages-scraper?

The yellow-pages-scraper targets Yellow Pages directory listings, which carry a different category taxonomy and may surface businesses missing from Google Maps. Using both gives you a cross-source view of a local market.

Does it handle reviews, not just listings?

The repository is positioned around listings extraction; full review-content scraping is a separate workflow with additional legal and ToS considerations. Treat any review-content pipeline as a distinct project with its own compliance review.

What happens when Google changes the page layout?

Field names and CSS hooks can change without warning. The expected response is to detect missing fields, update the repository or your selectors, redeploy, and rerun. Long-running pipelines need a continuous monitoring story.

Can I pipe the output into an AI agent or RAG pipeline?

Yes. JSON output loads cleanly into pandas, DuckDB, or a vector store. For agents that answer questions like "find plumbers rated above 4.5 in zip 78701 that are open right now", combine this scraper with a geocoder and a freshness check on opening hours.

Next Steps and Repository Links

Start by cloning the scrape-google-maps repository, installing its dependencies, and running a small batch against a single search query. Once the output schema matches your storage, scale up with proxy rotation and validate a steady cadence.

If your workflow depends on directory data, evaluate the yellow-pages-scraper for cross-source coverage. For a broader inventory of platform-specific scrapers covering YouTube, Zillow, Instagram, TikTok, and other sources, browse the data-scrape GitHub organization. Each repository follows the same configuration pattern: environment-based secrets, JSON output, and documented limits.