How to scrape Twitter (x.com) without authentication using Gaffa

This tutorial was written by John Fáwọlé and originally published on the Gaffa blog. I'm sharing it here with the DEV community.

Scraping a few tweets sounds simple until you hit a login prompt or your script returns a page without the posts you need. Because X (formerly Twitter) loads its content with JavaScript, a basic request often isn't enough, and you end up setting up a browser just to read a single post. With Gaffa's capture_dom action, you can load a public X profile in a real browser through one API call, with no login and no local browser. In this guide, you'll use Python and Gaffa's capture_dom to extract tweet text and engagement counts from real posts and save the results as JSON.

Can you scrape Twitter without an account?

Yes, you can scrape a public X post without logging in, but the page usually has to be rendered in a real browser first. A plain request may return a login page or incomplete content, whereas a browser runs the JavaScript needed to load the post.

X also offers an official API, but it requires an X account, developer credentials, and pay-per-use pricing with request limits, so it isn't an account-free option. Gaffa loads the public page for you without an X username, password, or saved session.

You're still limited to what X shows while signed out. A public profile may show only a few posts before asking you to log in, so the results may not include its full history.

Prerequisites

Before building the scraper, you'll need the following.

• Python 3.10 or newer
• A Gaffa API key
• A terminal and code editor
• The requests and beautifulsoup4 packages

You won't need to install Chrome, Selenium, or Playwright, since Gaffa handles the browser work for you.

How to scrape Twitter with Gaffa and Python

We'll build a scraper that loads a public X profile, finds the available tweet links, and saves each post's data as JSON. We'll use OpenAI's X account, but you can swap in any other public profile. You can follow along with the finished project in our GitHub repository.

Set up your Python project

Start by creating a project folder and a virtual environment to keep its packages separate from your other projects. On macOS or Linux, run the following.

mkdir twitter-scraper
cd twitter-scraper

python3 -m venv .venv
source .venv/bin/activate

On Windows PowerShell, use python instead of python3 and activate the environment with

.venv\Scripts\Activate.ps1

Next, install requests to make the API calls and beautifulsoup4 to read the HTML that Gaffa returns.

python -m pip install requests beautifulsoup4

Create a file named xscraper.py and add these imports and settings.

import getpass
import json
import os
import re
from datetime import datetime, timezone
from pathlib import Path
from urllib.parse import urlsplit

import requests
from bs4 import BeautifulSoup

PROFILE = "https://x.com/OpenAI"
LIMIT = 3
OUTPUT = Path("twitter_data")
API_KEY = (os.getenv("GAFFA_API_KEY") or getpass.getpass("Gaffa API key: ")).strip()

if not API_KEY:
    raise SystemExit("A Gaffa API key is required.")

OUTPUT.mkdir(exist_ok=True)

PROFILE is the page we'll scrape, while LIMIT caps the number of tweets we process, and we're starting with three so the first run stays small. The script reads your key from GAFFA_API_KEY or asks you to enter it, so you never have to expose it in the code.

Load a public X profile with Gaffa

We'll use capture_dom to get the page's rendered HTML and pass it to BeautifulSoup for extraction. Add the following function, which we'll reuse for the profile and for each tweet page.

def fetch_page(url, label):
    response = requests.post(
        "https://api.gaffa.dev/v1/browser/requests",
        headers={"X-API-Key": API_KEY},
        json={
            "url": url,
            "async": False,
            "proxy_location": "us",
            "settings": {
                "max_media_bandwidth": 1,
                "actions": [
                    {"type": "wait", "selector": "article", "timeout": 15000},
                    {"type": "capture_dom"},
                ],
            },
        },
        timeout=90,
    )
    (OUTPUT / f"{label}.json").write_text(response.text, encoding="utf-8")
    response.raise_for_status()
    body = response.json()
    data = body.get("data") or body
    actions = data.get("actions") or []

    if data.get("error") or not actions or any(a.get("error") for a in actions):
        raise RuntimeError(f"Gaffa could not load {url}. Check {label}.json.")
    if not 200 <= (data.get("http_status_code") or 0) < 300:
        raise RuntimeError(f"X returned HTTP {data.get('http_status_code')}.")

    html = requests.get(actions[-1]["output"], timeout=30)
    html.raise_for_status()
    html.encoding = "utf-8"
    (OUTPUT / f"{label}.html").write_text(html.text, encoding="utf-8")
    return BeautifulSoup(html.text, "html.parser")

The wait action holds the request for up to 15 seconds until an article element appears, which gives the content a chance to load before capture_dom runs. Setting async to False makes the call wait for its result, and we also use a US proxy and cap media downloads at 1 MB, since we only need text.

The final action returns a file URL for the captured HTML, so the function downloads it, saves a local copy, and returns a BeautifulSoup object. It also saves the raw API response so you can inspect any errors.

Now call the function to load the profile.

profile = fetch_page(PROFILE, "profile")

On a successful request, this creates twitter_data/profile.json and twitter_data/profile.html.

Find the available tweet URLs

With the profile loaded, we can look through its articles for tweet links that belong to the selected account.

profile_path = urlsplit(PROFILE).path.rstrip("/")
pattern = re.compile(rf"^{re.escape(profile_path)}/status/\d+$", re.IGNORECASE)
urls = []

for article in profile.select("article"):
    link = article.find("a", href=pattern)
    if link:
        url = "https://x.com" + link["href"]
        if url not in urls:
            urls.append(url)

urls = urls[:LIMIT]
if not urls:
    raise RuntimeError("No tweet links found. Check twitter_data/profile.html.")

print(f"Found {len(urls)} tweet URLs.")

This works with the HTML we already downloaded, so it makes no additional API requests. Keep in mind that LIMIT only sets a maximum, so raising it won't reveal posts that aren't in the captured page.

Extract tweet text, timestamps, and engagement counts

Next, we'll define how to read an individual tweet page. We identify the requested post by its timestamp link, which stops us from accidentally collecting a reply beneath it. First, add a small helper for reading engagement counts.

def get_count(article, label):
    element = article.select_one(f'[aria-label="{label}"]')
    return (element.get_text(strip=True) or None) if element else None

It returns the count as displayed on the page, or None when no count is available, so a value like "6.9K" stays as displayed instead of becoming an assumed exact number. Now add the function that extracts the tweet.

def extract_post(soup, url):
    path = urlsplit(url).path
    stamp = soup.select_one(f'article a[href="{path}"]')
    if stamp is None:
        raise ValueError("The requested post's timestamp link is missing.")

    article = stamp.find_parent("article")
    body = article.select_one('div[dir="auto"]')
    handle = path.split("/")[1]
    author = None
    author_path = re.compile(rf"^/{re.escape(handle)}$", re.IGNORECASE)
    for link in article.find_all("a", href=author_path):
        if link.get_text(strip=True).lower() == f"@{handle}".lower():
            author = link.get_text(strip=True)
            break
    if author is None or body is None or not body.get_text(strip=True):
        raise ValueError("The expected author or tweet text is missing.")
    if any(b.get_text(strip=True) == "Show more" for b in body.select("button")):
        raise ValueError("The tweet text is still truncated.")

    post_id = path.rsplit("/", 1)[-1]
    published = datetime.fromtimestamp(((int(post_id) >> 22) + 1288834974657) / 1000, timezone.utc)
    return {
        "post_id": post_id,
        "url": url,
        "author": author,
        "published_at": published.isoformat(timespec="seconds"),
        "text": body.get_text(" ", strip=True),
        "links": [a["href"] for a in body.select('a[href^="https://"]')],
        "likes_display": get_count(article, "Like"),
        "reposts_display": get_count(article, "Repost"),
        "replies_display": get_count(article, "Reply"),
    }

The function returns the tweet's ID, URL, author, publication time, text, links, and engagement counts. It also checks for missing content and a "Show more" button, so a shortened preview is never saved as a complete post.

Rather than parsing the date shown on the page, which changes with language and time zone, the function reads the publish time from the tweet ID itself. X builds every post ID from the moment it was created, so shifting the ID right by 22 bits and adding X's epoch (1288834974657 ms) gives the exact UTC timestamp, down to the second. The parser still depends on the HTML structure reflected in its selectors, so if extraction fails, compare those selectors with the saved HTML before changing the code.

Save the tweets as JSON

Now add the block that opens each URL with fetch_page(), extracts its data with extract_post(), and collects the results.

posts = []
failures = []

for url in urls:
    post_id = url.rsplit("/", 1)[-1]
    try:
        soup = fetch_page(url, post_id)
        post = extract_post(soup, url)
        posts.append(post)
        print(f"Extracted tweet {post_id}.")
    except (requests.RequestException, ValueError, RuntimeError, KeyError) as error:
        failures.append({"url": url, "error": str(error)})
        print(f"Could not extract {url}: {error}")

Each tweet is processed separately, so if a request or extraction fails, the script records the URL and error and carries on with the remaining posts. Finally, sort the successful records by date and write both lists to disk.

posts.sort(key=lambda post: post["published_at"], reverse=True)

(OUTPUT / "posts.json").write_text(
    json.dumps(posts, indent=2, ensure_ascii=False), encoding="utf-8"
)
(OUTPUT / "failures.json").write_text(
    json.dumps(failures, indent=2, ensure_ascii=False), encoding="utf-8"
)

print(f"Saved {len(posts)} tweets; {len(failures)} failed.")
if not posts:
    raise SystemExit("No tweets extracted. Check twitter_data/failures.json.")

posts.json contains the extracted tweets with the newest first, while failures.json lists any URLs that couldn't be processed along with their error messages. You can find the full code on our GitHub repository.

Run the scraper and check the results

Run the script from your project folder.

python xscraper.py

If it prompts for your Gaffa API key, paste it and press Enter. The terminal will report how many tweet URLs it found and whether each post was extracted, and once it finishes, open twitter_data/posts.json in your code editor to see the results.

The extracted tweets in posts.json, sorted with the newest post first.

Why use Gaffa instead of the X API?

The official X API is the better fit when you need supported, structured access to X data, but it needs developer credentials, comes with its own authentication per endpoint, and enforces strict usage limits. For a workflow like this one, Gaffa is simpler because it works with the public web page itself, handling JavaScript rendering, browser infrastructure, and proxy routing behind one REST API.

Scraping public tweets doesn't have to mean fighting login walls or managing a browser yourself. With Gaffa's capture_dom action, you can load a real X page, extract the data you need, and save it as clean JSON in a few lines of Python. Sign up for free to grab your 500 free credits and run your first request in minutes, or see how to scrape a table with Python for another take on the same pattern.

Frequently asked questions

How do I scrape Twitter without an X account using Gaffa?

Use capture_dom to load a public profile or tweet in Gaffa's browser, then parse the returned HTML with BeautifulSoup. No X username, password, or saved session is needed for posts visible while signed out.

Do I need an X API key to scrape tweets with Gaffa?

No. This method uses Gaffa's browser API instead of the official X API, so you only need a Gaffa API key to make the requests.

How do I scrape a different public X profile?

Change the PROFILE URL in the script to the account you want. Make sure its posts are visible while logged out, and check that the captured HTML still matches the selectors in extract_post().

How do I scrape X without Selenium or Playwright?

Send your request to Gaffa's /v1/browser/requests endpoint with wait and capture_dom actions. Gaffa runs the browser for you, so there's no Selenium, ChromeDriver, or local headless browser to install.

How do I get more than a few tweets from a profile?

Raising LIMIT only helps if the captured page contains more links. Signed-out profiles often show just a few posts, so the results are limited to what X makes available without a login.