Web scraping in 2026: Selenium vs Playwright vs httpx

#python #automation #webscraping

I've been working on web scraper toolkit for a while and wanted to share what I learned.

The problem

A flexible, production-ready web scraping framework that handles JavaScript-rendered pages, pagination, and anti-bot protections. Build scrapers for any website in minutes with built-in data cleaning and export tools.

What I built

Here are the main features I ended up shipping:

Supports static HTML and JavaScript-rendered pages
Built-in proxy rotation and user-agent spoofing
Automatic pagination and infinite scroll handling
Data cleaning and deduplication pipeline
Export to CSV, JSON, SQLite, or Excel

Code sample

# Basic structure
class Bot:
    def __init__(self, config):
        self.config = config

    def run(self):
        while True:
            self.scan()
            self.execute()