DEV Community

# webscraping

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
How to Block AI Web Scrapers on Your WordPress Site with robots.txt (2025 Update)

How to Block AI Web Scrapers on Your WordPress Site with robots.txt (2025 Update)

Comments
2 min read
Pinterest Scraping in 2025: Why I Built a Production-Ready Scrapy Spider (And You Should Too)

Pinterest Scraping in 2025: Why I Built a Production-Ready Scrapy Spider (And You Should Too)

Comments
4 min read
Data Extraction in Automated Workflows: The Competitive Edge

Data Extraction in Automated Workflows: The Competitive Edge

38
Comments 16
5 min read
Scraping Twitter in 2025: A Developer's Guide to Surviving the API Apocalypse

Scraping Twitter in 2025: A Developer's Guide to Surviving the API Apocalypse

5
Comments
3 min read
Building a Production-Ready Target Scraper: 5 Spiders That Actually Work in 2024

Building a Production-Ready Target Scraper: 5 Spiders That Actually Work in 2024

Comments
6 min read
What Is AI Scraping? A Modern Guide in 2025

What Is AI Scraping? A Modern Guide in 2025

Comments
8 min read
10 Best AI Web Scraping Tools of 2025

10 Best AI Web Scraping Tools of 2025

1
Comments
11 min read
BeautifulSoup vs lxml: A Practical Performance Comparison

BeautifulSoup vs lxml: A Practical Performance Comparison

2
Comments
3 min read
Proxies Alone Aren’t (Always) the Answer

Proxies Alone Aren’t (Always) the Answer

Comments
4 min read
🕸️ Web Scraping & Automation: How to Ethically Collect Web Data Without Getting Blocked (or Banned)

🕸️ Web Scraping & Automation: How to Ethically Collect Web Data Without Getting Blocked (or Banned)

5
Comments 3
3 min read
3 Subtle Mistakes That Slowed Down My Web App (And How I Fixed Them)

3 Subtle Mistakes That Slowed Down My Web App (And How I Fixed Them)

Comments
1 min read
5 best JavaScript web scraping libraries in 2025

5 best JavaScript web scraping libraries in 2025

1
Comments
7 min read
Beautiful Soup: Web Scraping's Delightful Deception

Beautiful Soup: Web Scraping's Delightful Deception

Comments
2 min read
🕷️For Developers Losing Hair Over Web Scraping: Is a Scraper API Your Lifesaver?

🕷️For Developers Losing Hair Over Web Scraping: Is a Scraper API Your Lifesaver?

2
Comments
2 min read
I Tested 25+ (Free & Paid) Web Scraping Tools - These 8 Are the Absolute Best

I Tested 25+ (Free & Paid) Web Scraping Tools - These 8 Are the Absolute Best

15
Comments
8 min read
🕸️ Web Scraping in Python: A Practical Guide for Data Scientists

🕸️ Web Scraping in Python: A Practical Guide for Data Scientists

1
Comments
3 min read
Using FireCrawl MCP Server with Claude for Web Scraping

Using FireCrawl MCP Server with Claude for Web Scraping

24
Comments 3
5 min read
How to scrape YouTube using Python [2025 guide]

How to scrape YouTube using Python [2025 guide]

7
Comments
18 min read
I Built a Node.js App That Tracks UK Supermarket Prices and Saved Me £300 This Month (Also My Neighbors Love It)

I Built a Node.js App That Tracks UK Supermarket Prices and Saved Me £300 This Month (Also My Neighbors Love It)

11
Comments
6 min read
Analysed over 500+ publicly available workflow templates

Analysed over 500+ publicly available workflow templates

Comments
1 min read
Building a YouTube Scraper with Scrapy: A Developer's Journey

Building a YouTube Scraper with Scrapy: A Developer's Journey

3
Comments 1
4 min read
How to Evade Web Scraping Bans

How to Evade Web Scraping Bans

Comments
5 min read
How I Scraped our Private Google Site in a Semi-Automated way

How I Scraped our Private Google Site in a Semi-Automated way

Comments
7 min read
From manual to autonomous: How AI-driven scraping saves 40% of time

From manual to autonomous: How AI-driven scraping saves 40% of time

55
Comments 2
9 min read
Building a Production-Ready LinkedIn Scraper with Python Scrapy 🐍

Building a Production-Ready LinkedIn Scraper with Python Scrapy 🐍

1
Comments
7 min read
loading...