DEV Community

# webscraping

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Using FireCrawl MCP Server with Claude for Web Scraping

Using FireCrawl MCP Server with Claude for Web Scraping

24
Comments 3
5 min read
How to scrape YouTube using Python [2025 guide]

How to scrape YouTube using Python [2025 guide]

7
Comments
18 min read
I Built a Node.js App That Tracks UK Supermarket Prices and Saved Me £300 This Month (Also My Neighbors Love It)

I Built a Node.js App That Tracks UK Supermarket Prices and Saved Me £300 This Month (Also My Neighbors Love It)

20
Comments 1
6 min read
Analysed over 500+ publicly available workflow templates

Analysed over 500+ publicly available workflow templates

Comments
1 min read
Building a YouTube Scraper with Scrapy: A Developer's Journey

Building a YouTube Scraper with Scrapy: A Developer's Journey

3
Comments 1
4 min read
How to Evade Web Scraping Bans

How to Evade Web Scraping Bans

Comments
5 min read
Scraping Twitter in 2025: A Developer's Guide to Surviving the API Apocalypse

Scraping Twitter in 2025: A Developer's Guide to Surviving the API Apocalypse

6
Comments
3 min read
How I Scraped our Private Google Site in a Semi-Automated way

How I Scraped our Private Google Site in a Semi-Automated way

Comments
7 min read
Building a Production-Ready LinkedIn Scraper with Python Scrapy 🐍

Building a Production-Ready LinkedIn Scraper with Python Scrapy 🐍

1
Comments 1
7 min read
I Tried 15+ Web Scraping Tools — Here's the Only One I'd Trust for Big Data Projects

I Tried 15+ Web Scraping Tools — Here's the Only One I'd Trust for Big Data Projects

10
Comments
7 min read
3 bewährte Wege für den HTML-zu-Excel-Export

3 bewährte Wege für den HTML-zu-Excel-Export

Comments 1
1 min read
From 500+ Lines to 150: How I Refactored My Amazon Scraper Into a Clean Scrapy Project

From 500+ Lines to 150: How I Refactored My Amazon Scraper Into a Clean Scrapy Project

Comments 2
4 min read
Mastering Residential Proxies with Python

Mastering Residential Proxies with Python

Comments
2 min read
AI-Ready Congressional Data: Building a Python Tool for Plaintext Extraction

AI-Ready Congressional Data: Building a Python Tool for Plaintext Extraction

3
Comments
5 min read
Web Scraping to SQL: How to Effectively Store and Analyze Your Scraped Data

Web Scraping to SQL: How to Effectively Store and Analyze Your Scraped Data

Comments
5 min read
Why your site needs a sitemap?

Why your site needs a sitemap?

31
Comments 4
3 min read
Google blocks Yandex: A Tale of Two robots.txt Files

Google blocks Yandex: A Tale of Two robots.txt Files

Comments
2 min read
Finding Hidden Comic Book Deals: A TypeScript Scraping Solution

Finding Hidden Comic Book Deals: A TypeScript Scraping Solution

1
Comments
6 min read
How to Block Requests with Puppeteer

How to Block Requests with Puppeteer

Comments
5 min read
How to Hide Cookie Banners with Puppeteer

How to Hide Cookie Banners with Puppeteer

6
Comments
3 min read
Supercharge Your Search Automation with Bright Data MCP + Google, Bing & Yandex

Supercharge Your Search Automation with Bright Data MCP + Google, Bing & Yandex

6
Comments
3 min read
Web Scraping Best Practices for CloudFront

Web Scraping Best Practices for CloudFront

Comments
5 min read
🧑‍💻 Web Scraping Indeed Remote Jobs and Storing in PostgreSQL

🧑‍💻 Web Scraping Indeed Remote Jobs and Storing in PostgreSQL

7
Comments 1
2 min read
How to Scrape Unlimited Google Maps Leads Using AI

How to Scrape Unlimited Google Maps Leads Using AI

3
Comments 1
9 min read
How to Scrape Amazon Product Data with Python

How to Scrape Amazon Product Data with Python

Comments
4 min read
loading...