DEV Community

Scrape Tweets from Twitter profiles without using Twitter's API.

Stefan Natter 🇦🇹👨🏻‍💻 on May 15, 2020

Have you ever wondered what's your most influential tweet since you created your Twitter profile? Or how your tweets performed in the last 30 days?...

Read full post

fitsum • May 17 '20 • Edited

this is SUPER cool. just a couple of issues with output

JSON output is invalid due to no commas between objects
db/sqlite3 option exports tables for followers, followers_names, following, following_names that are empty text and csv work as expected AFAICT

Stefan Natter 🇦🇹👨🏻‍💻 • May 17 '20

Hi Fitsum,

thanks for your feedback. Unfortunately, I am not a maintainer of twint. If you experience issues, report them on GitHub, please.

I'm sure they will be happy to help you and resolve the issues.

Burak Bayır • Aug 20

This was a solid Twint and Elasticsearch workflow when published, and the Docker setup makes the original architecture easy to understand. The important 2026 note is that Twint is no longer a reliable route against the current X platform. Several comments here already show the breakage readers encounter.

For a current implementation, Xquik provides profile, timeline, search, follower, reply, and tweet reads with cursor pagination and structured exports. That removes the need to keep a Twint fork, browser behavior, and an Elasticsearch ingestion layer working just to collect the source data. Elasticsearch can still be useful downstream if full-text analysis is the goal.

Xquik is managed, not a free open-source scraper, which is the main tradeoff. Pay-as-you-go starts at $10 and the credits do not expire. For recurring jobs, that can be a cheaper way to scrape tweets without the old Twitter API than maintaining the historical Twint stack.