Athreya aka Maneshwar

Posted on Jul 24

6 Open Source Tools That Give You the Web Back

#ai #webdev #productivity #programming

Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback.

You've probably noticed this if you've tried to fetch a public webpage in the last two years.

You write four lines of Python. You get a 403. You add a User-Agent header. You get a 403 with better manners.

You spin up headless Chrome and get a challenge page that spins forever.

Eventually you go looking at pricing pages for data APIs, where a cheerful landing page offers to sell you back the exact same public HTML for $100 a month.

Meanwhile the big labs vacuumed up that same web at a scale none of us will ever match, and now the standard way to read a page programmatically is to pay somebody.

The web went from "open by default" to "open, but only if you look like a person holding a mouse."

So let's talk about eight open source projects quietly clawing that access back. I've used most of these in anger. Some are brilliant.

Quick note on star counts: I mention a few, because they are a decent signal of "somebody else has hit the bugs before you." They are not a signal of maintenance. A repo can have 70k stars and a last commit from 2023. Always check the commit graph before you check the star count.

1. Firecrawl: the one that turned a URL into a prompt

What it does: you hand it a URL, it hands you clean markdown. Not HTML with the nav bar and the cookie banner and four newsletter modals. Markdown. Headings, paragraphs, links, done.

It also does more than single pages.

The crawl endpoint walks a whole site, map gives you a URL inventory without fetching everything, and structured extraction pulls typed JSON out of a page if you describe the shape you want.

from firecrawl import Firecrawl

app = Firecrawl(api_key="fc-...")
doc = app.scrape("https://example.com/docs/getting-started",
                 formats=["markdown"])
print(doc.markdown)

Reach for it when: you want the web as text, you want it now, and you would rather ship the feature than build a crawling team.

firecrawl / firecrawl

The API to search, scrape, and interact with the web at scale. 🔥

🔥 Firecrawl

The API to search, scrape, and interact with the web at scale. 🔥 The web context API to find sources, extract content, and turn it into clean Markdown or structured data your agents can ship with. Open source and available as a hosted service.

Pst. Hey, you, join our stargazers :)

Why Firecrawl?

Industry-leading reliability: Covers 96% of the web, including JS-heavy pages — no proxy headaches, just clean data (see benchmarks)
Blazingly fast: P95 latency of 3.4s across millions of pages, built for real-time agents and dynamic apps
LLM-ready output: Clean markdown, structured JSON, screenshots, and more — spend fewer tokens, build better AI apps
We handle the hard stuff: Rotating proxies, orchestration, rate limits, JS-blocked content, and more — zero configuration
Agent ready: Connect Firecrawl to any AI agent or MCP client with a single command
Media parsing…

View on GitHub

2. Crawl4AI: the same idea, but yours

What it does: an open source, async Python crawler that outputs LLM-ready markdown.

No API key. No credits. No rate limit other than the one your target site imposes and the one your conscience should.

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com")
        print(result.markdown)

asyncio.run(main())

Reach for it when: you have volume, you have infra people, or you have data you cannot legally hand to a third party service.

unclecode / crawl4ai

🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN

🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper.

🚀 Crawl4AI Cloud API — Closed Beta (Launching Soon)

Reliable, large-scale web extraction, now built to be drastically more cost-effective than any of the existing solutions.

👉 Apply here for early access
We’ll be onboarding in phases and working closely with early users Limited slots.

Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.

✨ Check out latest update v0.9.2

✨ New in v0.9.2: Maintenance patch release. Fixes a MemoryAdaptiveDispatcher task/page leak when a streaming crawl is closed, Docker Playground "Advanced Config" and Monitor WebSocket auth, Playwright headless-shell packaging, and GPU (ENABLE_GPU=true) Docker builds. Release notes →

✨ Recent v0.9.0: Major secure-by-default release of the Docker API server. Auth is on by default, the server binds loopback unless given a token, and the…

View on GitHub

Also, both of these projects have converged on the same insight, which is fun to watch.

3. Browser-use: an agent with hands

What it does: wires an LLM to a real browser and lets it drive. Clicking, typing, scrolling, filling forms, logging in, navigating a multi step checkout.

You describe the goal in English, it figures out the DOM.

Reach for it when: the target is genuinely interactive, the volume is low, and the alternative is a human doing it by hand.

browser-use / browser-use

🌐 Make websites accessible for AI agents. Automate tasks online with ease.

What can Browser Use do?

Browser Use lets an AI agent use a web browser the same way you do — it opens pages, clicks buttons, types, and fills in forms. You describe the task, and it completes it. For example, you can have it:

📋 Fill Forms

Task: "Fill in this job application with my resume and information."

Example code ↗

🍎 Extract data

Task: "Extract structured data about my followers and export it as a CSV."

extract-followers.mp4

Browser Use Cloud Docs ↗

💻 QA Automation

Task: "QA test my local website and report any bugs, usability issues, and visual inconsistencies."

Browser Use CLI ↗

Quickstart

If you want to use Browser Use in your agent (Claude Code, Codex, Cursor, Hermes, OpenClaw, etc.), paste this prompt, and it sets everything up itself:

Install or upgrade browser-use to the latest stable version with uv using Python 3.12, run `browser-use skill

…

View on GitHub

4. Scrapy: the one that was doing this before it was cool

What it does: the Python crawling framework. Request scheduling, concurrency, retries, middleware, item pipelines, autothrottle, the works.

It has been in production since roughly 2008 and it has crawled more pages than every other tool on this list combined.

Reach for it when: high volume, stable, mostly static targets. It is the boring correct answer and boring correct answers pay the bills.

scrapy / scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

Scrapy is a web scraping framework to extract structured data from websites. It is cross-platform, and requires Python 3.10+. It is maintained by Zyte (formerly Scrapinghub) and many other contributors.

Install with:

pip install scrapy

And follow the documentation to learn how to use it.

If you wish to contribute, see Contributing.

View on GitHub

5. Crawlee: the infrastructure you were about to rebuild badly

What it does: the Node.js (and now Python) crawling library from the Apify folks.

Proxy rotation, session management, browser fingerprint spoofing, automatic retries with backoff, request queues that survive a restart, and a unified API whether you are using plain HTTP or Playwright or Puppeteer under the hood.

Reach for it when: you are in the Node ecosystem, or your targets are actively hostile and you need the anti-blocking stack without paying for one.

apify / crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

A web scraping and browser automation library

Crawlee covers your crawling and scraping end-to-end and helps you build reliable scrapers. Fast.

Your crawlers will appear human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data, and store it to disk or cloud while staying configurable to suit your project's needs.

Crawlee is available as the crawlee NPM package.

👉 View full documentation, guides and examples on the Crawlee project website 👈

Do you prefer 🐍 Python instead of JavaScript? 👉 Checkout Crawlee for Python 👈.

Installation

We recommend visiting the Introduction tutorial in Crawlee documentation for more information.

Crawlee requires Node.js 16 or higher.

With Crawlee CLI

The fastest way to try Crawlee out is to use the Crawlee CLI and choose the Getting started example. The CLI will…

View on GitHub

6. Scrapling: the scraper that heals itself

What it does: a Python scraper that remembers what an element was, so when the site redesigns and your .product-title becomes .ProductCard__title--x7f2, it re-locates the element by similarity instead of returning an empty list.

Reach for it when: you maintain many spiders over a long time and selector rot is your actual bottleneck.

D4Vinci / Scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

Effortless Web Scraping for the Modern Web

Selection methods · Fetchers · Spiders · Proxy Rotation · CLI · MCP

Scrapling is an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl.

Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume and automatic proxy rotation - all in a few lines of Python. One library, zero compromises.

Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.

from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
StealthyFetcher.adaptive =

…

View on GitHub

About that "sell this as a service for $5,000 a month" thing

Every video on this topic ends with the same pitch: lead gen agencies charge thousands a month for this data, your cost is zero, go get rich.

I want to be careful here, because there is a real business in web data, but "your cost is zero" is doing an enormous amount of work in that sentence.

Your cost is not zero.

Your cost is residential proxies, which are the single biggest line item and which no open source project ships.

Your cost is CAPTCHA solving.

Your cost is the engineer-hours when four targets redesign in the same week.

Your cost is storage, monitoring, and the pager.

What those agencies are actually charging for is not the scraping code, which as you have just seen is free and excellent.

They are charging for the operational layer and the guarantee that the data shows up on Monday even though the site changed on Sunday.

That is a real service and it is worth real money.

Just go in knowing what you are actually selling.

The boring section that keeps you out of trouble

You knew this was coming.

Being able to scrape something is not the same as being allowed to, and the gap between those two is where people get their infrastructure banned or their company sued.

A short, non-lawyer summary of what I actually do:

Read robots.txt and honour it. Beyond ethics, it is self-interested: polite crawlers survive longer. Aggressive ones get IP ranges blocked, and then nobody on your team can scrape that domain.
Rate limit yourself before they rate limit you. Concurrency of 2 and a delay is almost always enough. Nobody needs 200 requests a second against a small site's origin server.
Check the Terms of Service, especially for anything behind a login. "Publicly accessible" and "you have permission" are different claims.
Personal data is a different legal universe. GDPR, CCPA and friends do not care that the data was on a public page. Scraping product prices and scraping names with email addresses are not the same activity.

Seriously. Open devtools, filter the network tab by Fetch/XHR, and look.

Roughly a third of the scrapers I have written could have been three lines against a JSON endpoint that was sitting there the whole time.

Scraping is what you do after that fails, not instead of checking.

Wrapping up

Six projects, one idea: the web is still readable, the tooling to read it is free and open, and most of the wall that showed up over the last two years is made of TLS fingerprints and inertia rather than actual locks.

Pick by job, not by star count. Start light. Check for an API. Be a polite guest on other people's servers.

And if you only take one thing from this: next time you get an inexplicable 403, before you reach for the browser, try impersonate="chrome". You will feel very silly and very happy.

What did I miss? I know some of you have a favourite obscure Go crawler you have been waiting for an excuse to talk about. Drop it in the comments, I read all of them.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs — without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

⭐ Star it on GitHub:

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

git-lrc

Free, Micro AI Code Reviews That Run on Commit

GenAI today is a race car without brakes. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub