The latest articles on DEV Community by Achyuth Jois M (@jois-code). https://dev.to/jois-code https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F458973%2F7a376598-8501-499e-93f2-05059e3af763.png https://dev.to/jois-code en Achyuth Jois M Tue, 13 Dec 2022 15:51:08 +0000 https://dev.to/jois-code/how-to-scrape-tweets-from-twitter-with-python-using-tweeds-4k2e https://dev.to/jois-code/how-to-scrape-tweets-from-twitter-with-python-using-tweeds-4k2e <h2> What is Tweeeds?🌱 </h2> <p>Tweeds is an advanced Twitter scraping tool written in Python that allows for scraping Tweets from Twitter. The name might be funny but <em>"Don't Judge a book by it's cover"</em>😏</p> <div class="ltag-github-readme-tag"> <div class="readme-overview"> <h2> <img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fassets.dev.to%2Fassets%2Fgithub-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"> <a href="https://github.com/jois-code" rel="noopener noreferrer"> jois-code </a> / <a href="https://github.com/jois-code/tweeds" rel="noopener noreferrer"> tweeds </a> </h2> <h3> An advanced Twitter scraping &amp; OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a Tweets and more while evading most API limitations. </h3> </div> <div class="ltag-github-body"> <div id="readme" class="md"> <div class="markdown-heading"> <h2 class="heading-element">Due to some changes in Twitter API interface we can't use this tool anymore!!</h2> </div> <div class="markdown-heading"> <h1 class="heading-element">Tweeds - Twitter Scraping Tool</h1> </div> <p> <a rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/249acc13e59e85e0deeb0faef7fe51077590e9b6fd177bc51d7fbb41d199d688/68747470733a2f2f6d656469612e74656e6f722e636f6d2f5872742d7479333950664541414141432f656c6f6e2d6d75736b2d736d6f6b652e676966"><img src="https://camo.githubusercontent.com/249acc13e59e85e0deeb0faef7fe51077590e9b6fd177bc51d7fbb41d199d688/68747470733a2f2f6d656469612e74656e6f722e636f6d2f5872742d7479333950664541414141432f656c6f6e2d6d75736b2d736d6f6b652e676966"></a> </p> <p>Scrape tweets from any Twitter user profile. Twitter API alternative to scrape Twitter hashtags, threads, images, videos, statistics and Twitter history. Export data in JSON and CSV format. This project enables you to extract large amounts of data from Twitter. It lets you do much more than the Twitter API, because it doesn't have rate limits and you don't even need to have a <strong>Twitter account, a registered app, or Twitter API key.</strong></p> <div class="markdown-heading"> <h3 class="heading-element">Do not forget to star this project.🌟 😍</h3> </div> <div class="markdown-heading"> <h2 class="heading-element">💡 Features</h2> </div> <ul> <li>No API Key required</li> <li>No Limit</li> <li>No Authentication required</li> <li>Get tweets from specific user</li> <li>JSON and CSV export for further Data Analysis</li> <li>And the best part it is OpenSource 😉</li> </ul> <div class="markdown-heading"> <h2 class="heading-element">✔️ Requirements</h2> </div> <ul> <li>Python &gt;= 3.9</li> </ul> <div class="markdown-heading"> <h2 class="heading-element">⚙ Installation</h2> </div> <div class="highlight highlight-source-shell notranslate position-relative overflow-auto js-code-highlight"> <pre>pip install tweeds</pre> </div> <div class="markdown-heading"> <h2 class="heading-element">💃 Usage</h2> </div> <div class="highlight highlight-source-shell notranslate position-relative overflow-auto js-code-highlight"> <pre>achyuthjoism-MBP <span class="pl-k">~</span> % tweeds ████████╗░██╗░░░░░░░██╗███████╗███████╗██████╗░░██████╗ ╚══██╔══╝░██║░░██╗░░██║██╔════╝██╔════╝██╔══██╗██╔════╝</pre>… </div> </div> </div> <div class="gh-btn-container"><a class="gh-btn" href="https://github.com/jois-code/tweeds" rel="noopener noreferrer">View on GitHub</a></div> </div> <h2> Why should I use Tweeds?🐐 </h2> <ol> <li>You don't need a Twitter API for scraping tweets</li> <li>You can scrape all the tweets(Twitter API limits 3000 tweets only😢!)</li> <li>No Twitter credentials required </li> <li>No limitation </li> <li>It's OpenSource 😃</li> </ol> <h2> Prerequisites ✅ </h2> <ul> <li>Python &gt;= 3.9</li> </ul> <h2> Installation ⚙ </h2> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>pip <span class="nb">install </span>tweeds </code></pre> </div> <p>And you are done! Now you can use this tool in your terminal or in your python project. In this post we will discuss both the use case.😎</p> <h2> Command Line Interface 💻 </h2> <ul> <li>Scrape all the tweets from Elon Musk(including replies). </li> </ul> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>tweeds <span class="nt">-u</span> elonmusk </code></pre> </div> <ul> <li>Scrape all Tweets from the Elon's timeline containing pineapple. </li> </ul> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>tweeds <span class="nt">-u</span> elonmusk <span class="nt">-s</span> pineapple </code></pre> </div> <ul> <li>Scrape all Tweets from user's tweeting about Coffee near Berlin. </li> </ul> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>tweeds <span class="nt">-s</span> Coffee <span class="nt">--near</span> Berlin </code></pre> </div> <ul> <li>Want to add a limit just add <code>--limit</code> argument. </li> </ul> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>tweeds <span class="nt">-s</span> Coffee <span class="nt">--near</span> Berlin <span class="nt">--limit</span> 1000 </code></pre> </div> <ul> <li>Scrape all Tweets from Elon's timeline since 2022-12-10. </li> </ul> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>tweeds <span class="nt">-u</span> elonmusk <span class="nt">--since</span> 2022-12-10 </code></pre> </div> <ul> <li>Scrape Tweets from a radius of 1km around a place in Paris. </li> </ul> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>tweeds <span class="nt">--geocode</span> <span class="s2">"48.880048,2.385939,1km"</span> <span class="nt">--limit</span> 10000 </code></pre> </div> <ul> <li>Want to export the data into CSV or JSON for further analysis just add <code>--csv</code> or <code>--json</code> argument with the filename. </li> </ul> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>tweeds <span class="nt">--geocode</span> <span class="s2">"48.880048,2.385939,1km"</span> <span class="nt">--csv</span> paris.csv <span class="nt">--limit</span> 10000 </code></pre> </div> <p>Refer <a href="https://github.com/achyuthjoism/tweeds#-example" rel="noopener noreferrer">Docs</a> for more example.</p> <h2> Modules ⚡ </h2> <p>Now Tweed can be used as a module and allows custom formatting. Example:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight python"><code><span class="kn">import</span> <span class="n">tweeds</span> <span class="n">query</span> <span class="o">=</span> <span class="n">tweeds</span><span class="p">.</span><span class="nc">Query</span><span class="p">()</span> <span class="n">query</span><span class="p">.</span><span class="n">search</span> <span class="o">=</span> <span class="sh">"</span><span class="s">Yash Boss</span><span class="sh">"</span> <span class="n">query</span><span class="p">.</span><span class="n">limit</span> <span class="o">=</span> <span class="mi">10</span> <span class="n">query</span><span class="p">.</span><span class="n">verified</span> <span class="o">=</span> <span class="bp">True</span> <span class="n">tweeds</span><span class="p">.</span><span class="nf">search</span><span class="p">(</span><span class="n">query</span><span class="p">)</span> </code></pre> </div> <p>Remember you have 20 options to fiddle with🤯.</p> <h2> Conclusion 👋 </h2> <p>This project was made only for educational purpose only. Thank you for following this tutorial to the end. Hopefully this article helps you, see you in the next article.<br> </p> <div class="ltag-github-readme-tag"> <div class="readme-overview"> <h2> <img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fassets.dev.to%2Fassets%2Fgithub-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"> <a href="https://github.com/jois-code" rel="noopener noreferrer"> jois-code </a> / <a href="https://github.com/jois-code/tweeds" rel="noopener noreferrer"> tweeds </a> </h2> <h3> An advanced Twitter scraping &amp; OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a Tweets and more while evading most API limitations. </h3> </div> <div class="ltag-github-body"> <div id="readme" class="md"> <div class="markdown-heading"> <h2 class="heading-element">Due to some changes in Twitter API interface we can't use this tool anymore!!</h2> </div> <div class="markdown-heading"> <h1 class="heading-element">Tweeds - Twitter Scraping Tool</h1> </div> <p> <a rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/249acc13e59e85e0deeb0faef7fe51077590e9b6fd177bc51d7fbb41d199d688/68747470733a2f2f6d656469612e74656e6f722e636f6d2f5872742d7479333950664541414141432f656c6f6e2d6d75736b2d736d6f6b652e676966"><img src="https://camo.githubusercontent.com/249acc13e59e85e0deeb0faef7fe51077590e9b6fd177bc51d7fbb41d199d688/68747470733a2f2f6d656469612e74656e6f722e636f6d2f5872742d7479333950664541414141432f656c6f6e2d6d75736b2d736d6f6b652e676966"></a> </p> <p>Scrape tweets from any Twitter user profile. Twitter API alternative to scrape Twitter hashtags, threads, images, videos, statistics and Twitter history. Export data in JSON and CSV format. This project enables you to extract large amounts of data from Twitter. It lets you do much more than the Twitter API, because it doesn't have rate limits and you don't even need to have a <strong>Twitter account, a registered app, or Twitter API key.</strong></p> <div class="markdown-heading"> <h3 class="heading-element">Do not forget to star this project.🌟 😍</h3> </div> <div class="markdown-heading"> <h2 class="heading-element">💡 Features</h2> </div> <ul> <li>No API Key required</li> <li>No Limit</li> <li>No Authentication required</li> <li>Get tweets from specific user</li> <li>JSON and CSV export for further Data Analysis</li> <li>And the best part it is OpenSource 😉</li> </ul> <div class="markdown-heading"> <h2 class="heading-element">✔️ Requirements</h2> </div> <ul> <li>Python &gt;= 3.9</li> </ul> <div class="markdown-heading"> <h2 class="heading-element">⚙ Installation</h2> </div> <div class="highlight highlight-source-shell notranslate position-relative overflow-auto js-code-highlight"> <pre>pip install tweeds</pre> </div> <div class="markdown-heading"> <h2 class="heading-element">💃 Usage</h2> </div> <div class="highlight highlight-source-shell notranslate position-relative overflow-auto js-code-highlight"> <pre>achyuthjoism-MBP <span class="pl-k">~</span> % tweeds ████████╗░██╗░░░░░░░██╗███████╗███████╗██████╗░░██████╗ ╚══██╔══╝░██║░░██╗░░██║██╔════╝██╔════╝██╔══██╗██╔════╝</pre>… </div> </div> </div> <div class="gh-btn-container"><a class="gh-btn" href="https://github.com/jois-code/tweeds" rel="noopener noreferrer">View on GitHub</a></div> </div> gratitude