X Scraper GitHub: How to Extract Public Posts and Profiles with Python

If you need a reproducible way to convert public X (formerly Twitter) posts and profiles into a structured dataset, the most practical starting point is an X scraper GitHub project that ships example code, a normalization layer, and a documented output schema. This article is for developers, data analysts, and research teams who want to build a public-post monitor from open-source Python rather than maintaining a full browser-automation stack themselves.

Quick answer

A repository-based X/Twitter scraper is a Python project that:

1. Loads a public post or profile URL and returns a normalized JSON record.
2. Keeps source URL and collection timestamp on every record so you can audit results later.
3. Lets you chain extraction, normalization, and storage as separate, testable steps.

The data-scrape GitHub organization hosts two relevant reference repositories for this workflow: the X Scraper project and the companion Twitter Scraper repository. Treat both as implementation references, not guaranteed production services. Always read the README, run the example with a small input set, and verify the output before relying on any field, endpoint, or claim.

Why public X/Twitter data is harder to monitor than it looks

Anyone building an X/Twitter monitor quickly runs into the same five constraints:

• Dynamic rendering. Post bodies, media URLs, and profile metadata are loaded by JavaScript after the initial HTML response. A plain HTTP client often receives an empty shell.
• Layout drift. CSS selectors, embedded JSON paths, and component structure change without warning. Extraction logic that worked last week may return null fields this week.
• Rate limiting and session behavior. Repeated requests from the same IP or user-agent pattern can trigger throttling, empty responses, or interactive challenges.
• Quota and access tiers. The official X API has tiered pricing and rate limits. A repository-based workflow is appealing precisely because it does not depend on an API key, but it inherits the anti-automation surface area of public web pages.
• Identity and duplication. The same post can be re-shared, the same handle can change display name, and retweet counts update independently of post body. Any monitor needs stable keys and de-duplication rules.

These constraints make self-hosted scraping viable for prototypes but expensive to maintain at scale. A repository-based workflow gives you a middle ground: you run the code, control the schema, and decide how much infrastructure to own.

What an X/Twitter scraper returns

A "scraper" in this context is a wrapper around the extraction layer. You pass a public post URL, a profile URL, or a search query, and the scraper returns normalized JSON. The exact fields depend on the repository implementation and the page type, but typical record shapes include:

Post-level fields

• post_id or tweet identifier
• author_handle and author_display_name
• post_text (the body text, cleaned)
• created_at (page-visible timestamp)
• language (when available)
• reply_count, repost_count, like_count, quote_count, view_count (page-visible counts)
• media_urls (first image or video URL when present)
• post_url (the canonical source URL)

Profile-level fields

• handle, display_name, bio
• follower_count, following_count
• post_count (tweets and replies combined)
• verified, profile_image_url
• profile_url (the canonical source URL)

Provenance

• collected_at: the time your workflow observed the page
• source: the public page origin
• review_status: a label for human validation

Because public pages differ by region, login state, and whether the account is verified or restricted, treat any field as optional until you have validated it against a controlled sample.

Step-by-step: building a public-post monitor

The workflow below assumes you have installed one of the repository-based tools and configured an endpoint or local runner according to its documentation. It uses environment variables for anything account-specific.

1. Set environment variables

export X_SCRAPER_ENDPOINT="https://your-endpoint-or-localhost.example.com/scrape"
export X_SCRAPER_API_KEY="your_api_key_if_required"
export X_OUTPUT_DIR="./data"

Replace the endpoint with the current value shown in the repository README or your own deployment.

2. Request a single public post

import json
import os
from datetime import datetime, timezone
from pathlib import Path

import requests

API_ENDPOINT = os.environ["X_SCRAPER_ENDPOINT"]
API_KEY = os.environ.get("X_SCRAPER_API_KEY")
OUTPUT_DIR = Path(os.environ.get("X_OUTPUT_DIR", "./data"))
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

headers = {"Content-Type": "application/json"}
if API_KEY:
    headers["Authorization"] = f"Bearer {API_KEY}"

# Example payload; confirm the exact schema in the repository README.
payload = {
    "url": "https://x.com/verge/status/1234567890123456789",
    "include_replies": False,
}

response = requests.post(API_ENDPOINT, headers=headers, json=payload, timeout=120)
response.raise_for_status()
raw_post = response.json()
print(json.dumps(raw_post, indent=2, ensure_ascii=False)[:1000])

3. Normalize the post record

def clean_text(value):
    if not isinstance(value, str):
        return None
    value = " ".join(value.split())
    return value or None


def as_positive_int(value):
    if isinstance(value, bool):
        return None
    if isinstance(value, int) and value > 0:
        return value
    if isinstance(value, str):
        digits = "".join(c for c in value if c.isdigit())
        if digits:
            return int(digits)
    return None


def normalize_post(raw, collected_at):
    post_id = clean_text(raw.get("post_id") or raw.get("id_str") or raw.get("id"))
    handle = clean_text(raw.get("author_handle") or raw.get("user_screen_name"))

    if not post_id or not handle:
        return None

    record = {
        "post_id": post_id,
        "author_handle": handle,
        "author_display_name": clean_text(
            raw.get("author_display_name") or raw.get("user_name")
        ),
        "post_text": clean_text(raw.get("post_text") or raw.get("text") or raw.get("full_text")),
        "created_at": clean_text(raw.get("created_at")),
        "language": clean_text(raw.get("lang")),
        "reply_count": as_positive_int(raw.get("reply_count")),
        "repost_count": as_positive_int(raw.get("repost_count") or raw.get("retweet_count")),
        "like_count": as_positive_int(raw.get("like_count") or raw.get("favorite_count")),
        "quote_count": as_positive_int(raw.get("quote_count")),
        "view_count": as_positive_int(raw.get("view_count")),
        "media_urls": raw.get("media_urls") or [],
        "post_url": clean_text(raw.get("post_url") or raw.get("url")),
        "collected_at": collected_at,
        "source": "public X post page",
        "review_status": "needs_review",
    }
    return record


collected_at = datetime.now(timezone.utc).isoformat()
records = []
post = normalize_post(raw_post, collected_at)
if post:
    records.append(post)

output_file = OUTPUT_DIR / f"post_{datetime.now(timezone.utc).strftime('%Y%m%d_%H%M%S')}.json"
output_file.write_text(json.dumps(records, indent=2, ensure_ascii=False), encoding="utf-8")
print(f"Saved {len(records)} normalized record(s) to {output_file}")

4. Normalize a profile record

def normalize_profile(raw, collected_at):
    handle = clean_text(raw.get("handle") or raw.get("screen_name"))
    if not handle:
        return None

    return {
        "handle": handle,
        "display_name": clean_text(raw.get("display_name") or raw.get("name")),
        "bio": clean_text(raw.get("bio") or raw.get("description")),
        "follower_count": as_positive_int(raw.get("follower_count")),
        "following_count": as_positive_int(raw.get("following_count")),
        "post_count": as_positive_int(raw.get("post_count") or raw.get("statuses_count")),
        "verified": bool(raw.get("verified")),
        "profile_image_url": clean_text(raw.get("profile_image_url")),
        "profile_url": clean_text(raw.get("profile_url") or f"https://x.com/{handle}"),
        "collected_at": collected_at,
        "source": "public X profile page",
        "review_status": "needs_review",
    }

This separation keeps extraction, normalization, and storage independent. If the extraction layer changes, you can update one function without rewriting the monitor logic.

Representative output shape

After normalization, a single post record looks like this:

{
  "post_id": "1234567890123456789",
  "author_handle": "verge",
  "author_display_name": "The Verge",
  "post_text": "Apple unveils new on-device language model for iOS 19.",
  "created_at": "2026-08-12T14:22:00.000Z",
  "language": "en",
  "reply_count": 84,
  "repost_count": 312,
  "like_count": 1502,
  "quote_count": 27,
  "view_count": 189000,
  "media_urls": [
    "https://pbs.twimg.com/media/Example123.jpg"
  ],
  "post_url": "https://x.com/verge/status/1234567890123456789",
  "collected_at": "2026-08-19T02:00:00+00:00",
  "source": "public X post page",
  "review_status": "needs_review"
}

A null field means the value was not present or not parseable in that run, not that it is zero or empty. Keeping that distinction visible prevents bad downstream decisions.

Use cases for a public-post monitor

News and trend tracking. Capture posts from a curated list of newsroom and analyst handles, then surface terms that spike in frequency or velocity.

Brand and reputation monitoring. Watch your brand name, product names, and competitor names, then route posts with high engagement to a human review queue.

Market research. Build a structured corpus of public posts around a topic for content analysis, sentiment scoring, or training data preparation.

Academic and policy research. Track a small set of public handles and hashtags with explicit consent and ethics review, then archive snapshots for later analysis.

Operations alerts. Notify a security or trust-and-safety team when specific keywords, handles, or media signatures appear on public posts.

Each use case should start with a narrow scope, clear data retention rules, and a manual review gate before any financial, legal, or personnel decision.

Repository workflow versus building from scratch

Dimension	Repository-based scraper	Build your own from scratch
Setup time	Hours to days, depending on the repo	Days to weeks for a reliable prototype
Maintenance	You update selectors and dependencies	You own the full extraction and anti-detection stack
Schema control	You define the normalization layer	You define everything
Proxy/headless overhead	Depends on repo design; may still need proxies	You manage browsers, proxies, and rotation
Field certainty	Validate from actual test outputs	Validate from your own parsing logic
Compliance	Your responsibility either way	Your responsibility either way

Neither option removes the obligation to respect X's terms, applicable law, and privacy requirements. A repository gives you a head start on code; it does not give you permission to collect or use data indiscriminately.

For a deeper look at structuring public-data projects and pipeline design choices that work across sources, see the Chinese-language public web data market research guide — it covers source selection, normalization, and the limits teams usually discover after the first pilot.

Limitations and compliance

• No official X API guarantee. Public pages are not a stable data contract. Fields, URLs, and availability can change.
• Regional variation. The same post may render differently depending on location, device fingerprint, and whether the viewer is logged in.
• Rate and volume limits. Running a monitor too aggressively can trigger blocks. Start with a small cadence and increase only after observing stable behavior.
• Data accuracy. Public posts can be edited, deleted, or restricted after you collect them. Always validate high-impact records before acting.
• Legal and platform terms. Use public data only for lawful purposes and in compliance with X's terms of service, robots directives where applicable, and applicable privacy laws. Do not evade access controls, logins, or rate limits, and do not collect data behind authentication walls.
• Personal data. If your records contain personal data, define retention, deletion, and access-control rules before you start collecting.

FAQ

Is the repository a managed production API?

No. The X Scraper repository and the Twitter Scraper repository are code references. Review their READMEs, dependencies, and recent commits before using them in any system.

What fields are guaranteed?

None. Public page extraction depends on the current page structure, region, login state, and account type. Validate every field you plan to use with a controlled test sample.

How often should the monitor run?

Start hourly for a small handle set, then move to daily for broader coverage. Increase frequency only if the decision you are supporting actually benefits from faster updates and you can handle the increased block risk.

Can this feed into a CRM, dashboard, or AI agent?

Yes, but only after normalization and review. The JSON output in the example above is designed to be stored in a database, sent to a webhook, or loaded as context for an AI agent. Keep provenance fields so the downstream system knows where each value came from.

What happens when X changes its page layout?

The extraction step may break or return partial records. A well-structured monitor detects increased null rates or parse failures and alerts a human to inspect the repository for updates.

Should I keep every raw response?

Only retain what your use case and governance rules justify. Keep enough to audit a result, but do not accumulate unnecessary personal or sensitive content.

Where do I inspect the referenced projects?

Start with the data-scrape GitHub profile, then review the X Scraper repository and the Twitter Scraper repository.

Next step

Before running a production monitor, define the exact handles, hashtags, or search queries you need, write a target JSON schema, collect a small test sample, and assign someone to validate the output. The repositories above are useful starting points for that evaluation; the durable asset is the documented, reviewable workflow your team builds around them.