DEV Community

# webscraping

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
How I Built a WhatsApp Ad Leads Finder with Apify

How I Built a WhatsApp Ad Leads Finder with Apify

Comments 1
2 min read
Scraping eBay's 1.2 Billion Listings: A Scrapy Tale

Scraping eBay's 1.2 Billion Listings: A Scrapy Tale

Comments
4 min read
A Faster Approach to Email Validation: Why We Ditched Regex

A Faster Approach to Email Validation: Why We Ditched Regex

Comments
3 min read
How to Block AI Web Scrapers on Your WordPress Site with robots.txt (2025 Update)

How to Block AI Web Scrapers on Your WordPress Site with robots.txt (2025 Update)

Comments
2 min read
Pinterest Scraping in 2025: Why I Built a Production-Ready Scrapy Spider (And You Should Too)

Pinterest Scraping in 2025: Why I Built a Production-Ready Scrapy Spider (And You Should Too)

Comments
4 min read
Data Extraction in Automated Workflows: The Competitive Edge

Data Extraction in Automated Workflows: The Competitive Edge

38
Comments 16
5 min read
Scraping Twitter in 2025: A Developer's Guide to Surviving the API Apocalypse

Scraping Twitter in 2025: A Developer's Guide to Surviving the API Apocalypse

5
Comments
3 min read
Building a Production-Ready Target Scraper: 5 Spiders That Actually Work in 2024

Building a Production-Ready Target Scraper: 5 Spiders That Actually Work in 2024

Comments
6 min read
What Is AI Scraping? A Modern Guide in 2025

What Is AI Scraping? A Modern Guide in 2025

Comments
8 min read
10 Best AI Web Scraping Tools of 2025

10 Best AI Web Scraping Tools of 2025

1
Comments
11 min read
BeautifulSoup vs lxml: A Practical Performance Comparison

BeautifulSoup vs lxml: A Practical Performance Comparison

2
Comments
3 min read
Proxies Alone Aren’t (Always) the Answer

Proxies Alone Aren’t (Always) the Answer

Comments
4 min read
🕸️ Web Scraping & Automation: How to Ethically Collect Web Data Without Getting Blocked (or Banned)

🕸️ Web Scraping & Automation: How to Ethically Collect Web Data Without Getting Blocked (or Banned)

5
Comments 3
3 min read
3 Subtle Mistakes That Slowed Down My Web App (And How I Fixed Them)

3 Subtle Mistakes That Slowed Down My Web App (And How I Fixed Them)

Comments
1 min read
Mastering Residential Proxies with Python

Mastering Residential Proxies with Python

Comments
2 min read
5 best JavaScript web scraping libraries in 2025

5 best JavaScript web scraping libraries in 2025

1
Comments
7 min read
Beautiful Soup: Web Scraping's Delightful Deception

Beautiful Soup: Web Scraping's Delightful Deception

Comments
2 min read
🕷️For Developers Losing Hair Over Web Scraping: Is a Scraper API Your Lifesaver?

🕷️For Developers Losing Hair Over Web Scraping: Is a Scraper API Your Lifesaver?

2
Comments
2 min read
I Tested 25+ (Free & Paid) Web Scraping Tools - These 8 Are the Absolute Best

I Tested 25+ (Free & Paid) Web Scraping Tools - These 8 Are the Absolute Best

15
Comments
8 min read
🕸️ Web Scraping in Python: A Practical Guide for Data Scientists

🕸️ Web Scraping in Python: A Practical Guide for Data Scientists

1
Comments
3 min read
Using FireCrawl MCP Server with Claude for Web Scraping

Using FireCrawl MCP Server with Claude for Web Scraping

24
Comments 3
5 min read
How to scrape YouTube using Python [2025 guide]

How to scrape YouTube using Python [2025 guide]

7
Comments
18 min read
I Built a Node.js App That Tracks UK Supermarket Prices and Saved Me £300 This Month (Also My Neighbors Love It)

I Built a Node.js App That Tracks UK Supermarket Prices and Saved Me £300 This Month (Also My Neighbors Love It)

11
Comments
6 min read
Analysed over 500+ publicly available workflow templates

Analysed over 500+ publicly available workflow templates

Comments
1 min read
Building a YouTube Scraper with Scrapy: A Developer's Journey

Building a YouTube Scraper with Scrapy: A Developer's Journey

3
Comments 1
4 min read
loading...