How to Scrape X (Twitter) in 2026 — Why Datacenter Scrapers Keep Dying

#datascience #webscraping #python #twitter

TL;DR: X blocks search from every datacenter IP range. Your scraper isn't broken — your IP is. Here are the 4 approaches that still work, with code.

If you've built a Twitter scraper in the last few months, you've probably seen this: same code, worked yesterday, 404 Not Found today. No error message. No rate-limit headers. Just 404s.

It's not your code. X (Twitter) returns 404 for SearchTimeline requests from virtually every datacenter IP range — AWS, GCP, Azure, Hetzner, OVH, and even carrier CGNAT ranges. The endpoint works fine from a residential ISP and fails from a VPS with the exact same headers, cookies, and TLS fingerprint.

Let me walk through what still works in 2026, from easiest to hardest.

Option 1: Profile data — free, no browser, no login

Here's something most people miss: x.com serves profile pages server-rendered to logged-out visitors. No API key, no headless browser, no cookies. A plain HTTP GET returns the full profile in the HTML:

import requests, re

def get_profile(username):
    r = requests.get(
        f"https://x.com/{username}",
        headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"},
    )
    html = r.text
    anchor = re.search(rf'screenName:"{username}"', html)
    window = html[anchor.start() - 1200 : anchor.end() + 1500]
    return {
        "bio": re.search(r'description:"((?:[^"\\]|\\.)*)"', window).group(1),
        "followers": int(re.search(r"followers:(\d+)", window).group(1)),
        "following": int(re.search(r"following:(\d+)", window).group(1)),
        "tweets": int(re.search(r"tweets:(\d+)", window).group(1)),
        "user_id": re.search(r'restId:"(\d+)"', window).group(1),
    }

print(get_profile("elonmusk"))

Exact counts, not the rounded "8M" you get from the UI. About 2 seconds per profile. This works from datacenter IPs because it's just the regular public page.

Option 2: Managed scrapers (Apify, pay-per-result)

If you don't want to maintain scraping infrastructure, Apify has X scrapers where you pay per result — you don't pay for failed runs.

The pricing spread is interesting:

Scraper	Price / 1K tweets	Approach	Uptime
apidojo X Scraper	$0.40/1K	datacenter	breaks periodically
xquik	$0.15/1K	datacenter	breaks periodically
Ghostfox X Scraper	$0.10/1K	real device	100% since launch

The device-based one (autokeren/ghostfox-x-scraper) is the approach I'd pick if I were buying instead of building — it routes search through an actual phone with a real ISP connection, which is the one thing X doesn't block. It also handles the profile scraping from Option 1 as a bonus mode.

Python usage via their free client:

from xscraper import search, profile

tweets = search("machine learning", max_tweets=20, tab="live")
p = profile("openai")[0]

New Apify accounts get $5 free credit, which at $0.10/1K is 50K tweets — enough to decide if it works for you.

Option 3: Roll your own device bridge

The DIY version of the above. You need:

A spare Android phone with a real SIM/mobile data
An accessibility-based automation framework (uiautomator2, or a custom AccessibilityService — screenscraping via adb exec-out screencap is too brittle)
A tunnel to expose it (cloudflared works fine)

Your scraper then: opens the app → deep-links to x.com/search?q=... → scrolls → reads the accessibility tree → parses tweets. It's the same data the UI shows, so no API, no anti-bot. The downside: one phone = one concurrent search, and you're now maintaining hardware.

Option 4: Residential proxies + regular scraping

The classic approach. It still works but it's a treadmill: proxy quality degrades, X tightens, you pay more. Expect $1–5/GB for residential proxies that survive, and budget for the arms race.

What about the official API?

At $100/month for basic tier (10K tweets read), it's only worth it if you're building something commercial with predictable volume. For research and scraping-style workloads, the pricing doesn't fit — which is why everyone's here reading about scrapers.

Bottom line

Profiles → just HTTP, no anti-scraping exists for this, use the code above
Search → device-based is the only approach with real uptime; managed device scrapers exist from $0.10/1K
DIY → fun weekend project, painful long-term maintenance

Happy to answer questions in the comments.

DEV Community