Twitter Scraper GitHub: How to Build a Public Mention and Sentiment Dataset with Python

The fastest way to track what people are saying about a brand or topic on X/Twitter without committing to a paid API tier is to collect public posts with an open-source scraper repository and add a small Python analysis layer. This article is for growth marketers, brand analysts, social-listening engineers, and AI-agent builders who need a reproducible mention-and-sentiment dataset built from public posts.

The data-scrape/twitter-scraper repository provides a runnable Python reference for public X/Twitter posts, and the same normalization pattern works for data-scrape/x-scraper and data-scrape/instagram-post-scraper, so the sentiment pipeline you build here can be extended across public social platforms.

TL;DR

• Use data-scrape/twitter-scraper to collect public X/Twitter post records.
• Define a narrow watchlist up front: target handles, hashtags, or keyword phrases. Broad queries produce noisy, hard-to-maintain datasets.
• Normalize each raw post into a stable schema with captured_at, source URL, author, text, engagement metrics, and extracted entities.
• Score sentiment with a local, rule-based lexicon so you do not need an external NLP API or fine-tuned model to start.
• Aggregate results by hour, day, and target entity so trend direction is easy to read.
• Refresh daily for brand monitoring; refresh hourly only during active campaigns or incidents, and collect only public, non-authenticated posts.

Why Public Mention and Sentiment Is Harder Than It Looks

Public social feeds look simple until you try to turn them into a reliable dataset. A handful of problems recur:

• Official API access is gated and expensive. The X API requires paid tiers for most search use cases, so many teams prefer to own a lightweight mention pipeline.
• HTML is JavaScript-heavy. A basic requests.get() call usually returns a hydration shell, not the visible post text, so you need a browser-automation or embedded-data approach.
• Posts are noisy. Hashtags, cashtags, retweets, quote tweets, replies, and threads have different structures, and a brand-name query returns irrelevant matches unless you filter.
• Sentiment is context-dependent. A simple word list misclassifies sarcasm, slang, and emoji, so the goal is a directional signal, not ground truth.
• Rate limits and layout drift. Aggressive polling hits throttling fast, and selectors break when the platform updates its front end.

This article takes the middle path: start with an open-source reference repository, then add a small, auditable Python layer for normalization and sentiment.

What the Verified Repositories Provide

The data-scrape/twitter-scraper repository is a Python reference for collecting public X/Twitter post records. It ships a scraper.py entry point, a requirements.txt, an examples/ folder, and a README. Read the README for the current setup steps and environment variables.

The same pattern appears across the data-scrape profile: data-scrape/x-scraper returns public post and profile records from the same platform, and data-scrape/instagram-post-scraper returns public post records whose captions and comments can be fed into the same normalization layer. The repositories differ in raw field names but share enough shape that one script can be reused across all three platforms.

Pipeline Design

A maintainable pipeline has five stages: capture raw JSON against a bounded watchlist, normalize fields into a stable schema, extract mentions and hashtags, score sentiment with a local lexicon, and aggregate per-post scores into time-window summaries by entity.

Setup and Configuration

Clone the repository and install dependencies inside a virtual environment so packages do not collide with system Python:

git clone https://github.com/data-scrape/twitter-scraper.git
cd twitter-scraper
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Set configuration through environment variables instead of hard-coding queries or pacing limits. The example below forwards a target list, output path, and request budget to the scraper's CLI. Adjust variable names to match the current version of scraper.py; never commit a real query list or proxy configuration to source control.

export TWITTER_QUERIES_FILE="watchlist.txt"
export TWITTER_OUTPUT_FILE="raw_posts.json"
export TWITTER_MAX_REQUESTS=300
python scraper.py \
  --input "$TWITTER_QUERIES_FILE" \
  --output "$TWITTER_OUTPUT_FILE" \
  --max-requests "$TWITTER_MAX_REQUESTS"

Keep watchlist.txt narrow: one brand handle, two competitor handles, and a couple of campaign hashtags is usually enough to start.

Normalization: From Raw JSON to a Stable Schema

The script below reads raw_posts.json, normalizes each record, extracts mentions and hashtags, scores sentiment with a tiny local lexicon, and writes the result to JSONL. Treat the score as a directional signal, not a definitive emotion label.

import json
import pathlib
import os
import re
import sys
from collections import Counter
from datetime import datetime, timezone


RAW_PATH = pathlib.Path(os.environ.get("TWITTER_RAW_PATH", "raw_posts.json"))
OUT_PATH = pathlib.Path(os.environ.get("TWITTER_OUT_PATH", "normalized_posts.jsonl"))
ROLLUP_PATH = pathlib.Path(os.environ.get("TWITTER_ROLLUP_PATH", "sentiment_rollup.json"))
CAPTURED_AT = datetime.now(timezone.utc).isoformat()

# Minimal lexicon for illustration. Expand it with domain-specific words.
POSITIVE_WORDS = {
    "love", "great", "good", "awesome", "excellent", "happy", "best",
    "amazing", "fantastic", "recommend", "thanks", "thank", "helpful",
    "solid", "smooth", "clean", "fast", "easy", "useful",
}
NEGATIVE_WORDS = {
    "hate", "bad", "terrible", "worst", "broken", "slow", "awful",
    "disappointed", "frustrating", "useless", "bug", "crash", "poor",
    "expensive", "fail", "failed", "annoying", "waste",
}


def to_int(value):
    """Coerce engagement fields into integers."""
    if value is None:
        return None
    try:
        return int(str(value).replace(",", "").split(".")[0])
    except ValueError:
        return None


def extract_entities(text: str) -> dict:
    """Extract mentions, hashtags, and cashtags from a post body."""
    return {
        "mentions": re.findall(r"@(\w+)", text),
        "hashtags": re.findall(r"#(\w+)", text),
        "cashtags": re.findall(r"\$(\w+)", text),
    }


def sentiment_scores(text: str) -> dict:
    """Return rule-based sentiment proportions and a compound score."""
    tokens = re.findall(r"\b\w+\b", text.lower())
    pos = sum(1 for t in tokens if t in POSITIVE_WORDS)
    neg = sum(1 for t in tokens if t in NEGATIVE_WORDS)
    total = len(tokens) or 1
    neu = total - pos - neg
    compound = round((pos - neg) / total, 4)
    return {
        "positive": pos,
        "negative": neg,
        "neutral": max(neu, 0),
        "compound": compound,
    }


def normalize(raw: dict) -> dict:
    text = raw.get("text") or raw.get("content") or raw.get("body") or ""
    metrics = raw.get("metrics") or raw.get("engagement") or {}
    entities = extract_entities(text)

    return {
        "platform": "x-twitter",
        "post_id": raw.get("id") or raw.get("tweet_id") or raw.get("status_id"),
        "author": raw.get("author") or raw.get("username") or raw.get("user_handle"),
        "posted_at": raw.get("posted_at") or raw.get("timestamp") or raw.get("created_at"),
        "captured_at": raw.get("captured_at") or CAPTURED_AT,
        "source_url": raw.get("source_url") or raw.get("url"),
        "text": text,
        "metrics": {
            "likes": to_int(metrics.get("likes") or metrics.get("like_count")),
            "replies": to_int(metrics.get("replies") or metrics.get("reply_count")),
            "retweets": to_int(metrics.get("retweets") or metrics.get("retweet_count")),
            "quotes": to_int(metrics.get("quotes") or metrics.get("quote_count")),
        },
        "entities": entities,
        "sentiment": sentiment_scores(text),
        "metadata": {
            "category": "Social Scrapers",
            "query": raw.get("metadata", {}).get("query"),
        },
    }


def rollup(records: list) -> dict:
    """Aggregate sentiment and volume by entity and day."""
    by_entity = {}
    for record in records:
        day = record.get("posted_at", record.get("captured_at", ""))[:10] or "unknown"
        for entity_type in ("mentions", "hashtags"):
            for entity in record["entities"].get(entity_type, []):
                key = (entity_type, entity.lower(), day)
                bucket = by_entity.setdefault(key, {
                    "entity_type": entity_type,
                    "entity": entity.lower(),
                    "day": day,
                    "posts": 0,
                    "compound_sum": 0.0,
                    "likes": 0,
                    "positive": 0,
                    "negative": 0,
                })
                bucket["posts"] += 1
                bucket["compound_sum"] += record["sentiment"]["compound"]
                bucket["likes"] += record["metrics"]["likes"] or 0
                bucket["positive"] += record["sentiment"]["positive"]
                bucket["negative"] += record["sentiment"]["negative"]

    result = []
    for bucket in by_entity.values():
        bucket["avg_compound"] = round(bucket["compound_sum"] / bucket["posts"], 4) if bucket["posts"] else 0
        result.append(bucket)
    return result


def main() -> int:
    if not RAW_PATH.is_file():
        print(f"Raw file not found: {RAW_PATH}", file=sys.stderr)
        return 1

    raw_data = json.loads(RAW_PATH.read_text(encoding="utf-8"))
    records = raw_data if isinstance(raw_data, list) else [raw_data]

    normalized = [normalize(r) for r in records]

    OUT_PATH.parent.mkdir(parents=True, exist_ok=True)
    with OUT_PATH.open("w", encoding="utf-8") as fp:
        for record in normalized:
            fp.write(json.dumps(record, ensure_ascii=False) + "\n")

    summary = rollup(normalized)
    ROLLUP_PATH.write_text(json.dumps(summary, indent=2, ensure_ascii=False), encoding="utf-8")

    print(f"Wrote {len(normalized)} normalized records to {OUT_PATH}")
    print(f"Wrote {len(summary)} entity-day rollups to {ROLLUP_PATH}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

The script is deliberately dependency-light. If you later need higher accuracy, replace the sentiment_scores function with a library such as VADER or a hosted model, but keep the same output shape so downstream aggregation code does not change.

Representative Output

Each line of the JSONL file looks like this:

{
  "platform": "x-twitter",
  "post_id": "1234567890123456789",
  "author": "example_user",
  "posted_at": "2026-09-08T08:30:00+00:00",
  "captured_at": "2026-09-08T10:00:00+00:00",
  "source_url": "https://x.com/example_user/status/1234567890123456789",
  "text": "Just tried the new update — smooth, fast, and exactly what I needed. Great work @examplebrand! #opensource",
  "metrics": {
    "likes": 42,
    "replies": 3,
    "retweets": 8,
    "quotes": 1
  },
  "entities": {
    "mentions": ["examplebrand"],
    "hashtags": ["opensource"],
    "cashtags": []
  },
  "sentiment": {
    "positive": 2,
    "negative": 0,
    "neutral": 16,
    "compound": 0.1111
  },
  "metadata": {
    "category": "Social Scrapers",
    "query": "examplebrand"
  }
}

Treat this as a representative shape, not a guaranteed output. Read the actual raw_posts.json before assuming field names; the X/Twitter schema changes when the platform updates its front end.

Use Cases

A normalized dataset supports several workflows:

• Brand health tracking. Plot average compound sentiment per day for your handle and selected competitors.
• Campaign measurement. Compare volume and sentiment before, during, and after a product launch.
• Incident detection. Watch for sudden spikes in negative mentions and replies with threshold alerts.
• AI-agent context. Feed the JSONL into a retriever so an agent can answer questions like "What did people say about our pricing yesterday?" without scraping live pages.
• Cross-platform rollups. Reuse the script with data-scrape/x-scraper and data-scrape/instagram-post-scraper to compare X/Twitter and Instagram conversations side by side.

Comparison: Official API, Open-Source Scraper, and Hosted Listening Platform

Three common sourcing options:

Dimension	Official X API	Open-source repo (e.g., data-scrape/twitter-scraper)	Hosted social-listening platform
Best for	Teams with budget and compliance requirements that need high-volume, real-time streams	Teams that want a runnable reference and accept responsibility for hosting, scheduling, and schema maintenance	Teams that want dashboards, alerts, and reports without building infrastructure
Setup model	Apply for access, choose a paid tier, authenticate, and call the documented endpoints	Clone the repo, install requirements, configure the CLI, schedule refreshes	Sign up, connect accounts or search terms, and configure dashboards
Output format	JSON via API; check current docs for field names and rate limits	JSON or CSV from the scraper, plus your own normalized JSONL	Dashboard widgets, CSV export, or API; varies by vendor
Maintenance burden	Low once integrated; policy and pricing changes are vendor-managed	Medium: scheduling, retries, schema drift, proxy policy, and layout changes	Low: vendor maintains the integration, but export formats may change
Sentiment analysis	Not provided; you supply your own NLP layer	You supply and control the lexicon or model	Usually included; verify accuracy and customization limits
Pricing verification	Official X developer pricing page	Free open source; you pay for your own hosting and proxies	The vendor's current pricing page; do not trust third-party summaries

Operational Checklist

Before you run a mention pipeline in any non-trivial environment, walk through this checklist:

• [ ] Read X/Twitter's current terms of service, developer policy, and any jurisdiction-specific platform guidelines.
• [ ] Your watchlist is narrow enough to return useful results without hammering the platform.
• [ ] You set request pacing (delays, jitter) to avoid rate limits or CAPTCHA walls.
• [ ] You capture captured_at, the source URL, and the raw text block on every record.
• [ ] Your sentiment lexicon is documented and versioned, and you understand its known biases.
• [ ] You have a dedupe strategy based on post_id or source_url, not on text content.
• [ ] You have a backoff and retry policy and you log the failures.
• [ ] You have a refresh cadence that matches your decision speed; daily is enough for most brand monitoring.
• [ ] You have reviewed applicable privacy and data-protection laws for the records you collect.

Limits, Maintenance, and Compliance

X/Twitter's HTML, JavaScript, and anti-automation measures evolve, and the repository's README reflects a snapshot of that evolution. The repositories do not ship a hosted proxy, a CAPTCHA solver, or an unlimited request budget; you provide those, or you accept a smaller, slower dataset.

Public X/Twitter data still touches trademark, database, and consumer-protection law in many jurisdictions, and the platform's terms restrict certain automated access. Restrict your collection to public, non-authenticated posts, respect robots directives and the platform's terms, do not attempt to bypass access controls, and do not republish records in misleading ways. Sentiment scores from a small lexicon are a directional signal, not a psychological measurement; always label the method when you share a chart or a number.

FAQ

Is there an official X/Twitter API I can use instead?
Yes. The X API offers search and streaming endpoints, but most useful search access requires a paid tier. Verify current tiers, pricing, and terms on the official X developer documentation page. This article covers the open-source, self-managed path for teams that do not have or do not want API access.

What data fields does the repository return?
Field names depend on the current version of scraper.py and the query. Treat the example record as a representative shape, not a guarantee. Read raw_posts.json and the README before integrating.

How often should the workflow run?
Daily refreshes are sufficient for most brand-health monitoring. Hourly refreshes are reasonable during active campaigns or incidents. Avoid sub-hourly refreshes without a documented reason and a sustainable request budget.

What happens when the page layout changes?
Layout changes break CSS selectors and JSON keys. Keep a small set of "smoke test" posts that fail loudly if a core field disappears; treat a missing text or author field as a hard error.

How do I make the sentiment scoring more accurate?
Replace the small lexicon in sentiment_scores with a domain-specific lexicon, VADER, TextBlob, or a fine-tuned model. Keep the same output keys (positive, negative, neutral, compound) so the rollup logic stays unchanged.

Can this connect to a dashboard, CRM, or AI agent?
Yes. The JSONL output can be loaded by pandas, DuckDB, or a vector store. Emit the rollup file to a scheduled job that writes to Google Sheets, a Slack channel, or a CRM activity feed. For AI agents, expose the JSONL records through a retriever so the agent can answer questions about mention trends without browsing live pages.

What should be verified before production use?
Confirm that your watchlist is narrow, your request pacing respects platform limits, your proxy or IP rotation policy is documented, your sentiment method is calibrated against a sample of manually labeled posts, and your data retention policy complies with applicable law.

Next Steps

Clone the data-scrape/twitter-scraper repository, follow the README, and start with the normalization script. Add data-scrape/x-scraper and data-scrape/instagram-post-scraper when you want to compare X/Twitter and Instagram conversations on the same schema. The rest of the data-scrape profile holds adjacent open-source repositories you can adopt as your pipeline grows.

Related Articles

• X Scraper GitHub: How to Extract Public Posts and Profiles with Python
• TikTok Scraper GitHub: How to Build a Public Video and Creator Pipeline with Python
• Instagram Post Scraper GitHub: How to Calculate Public Engagement Rate with Python