Web scraping is one of those tasks that sounds simple until you actually try it. You hit a site, grab the text, and move on — right?
Then reality hits: dynamic JavaScript rendering, bot detection, messy HTML, paywalls, cookie banners, and pages that change their layout every other week. Suddenly your "simple" scraper is a full-time maintenance job.
In this article, I compare five common ways to extract content from web pages in 2026, including a dedicated Web to Markdown/JSON API that removes most of the pain. I'll show real code for each so you can pick the right tool for your project.
The Five Approaches
| Approach | Setup Effort | Handles JS? | Maintenance | Best For |
|---|---|---|---|---|
1. requests + BeautifulSoup |
Low | ❌ No | High | Static pages you fully control |
| 2. Playwright / Selenium | High | ✅ Yes | High | JS-heavy apps, scraping behind logins |
| 3. Trafilatura / newspaper3k | Low | ❌ No | Medium | Article extraction from raw HTML |
| 4. A dedicated extraction API | None | ✅ Yes | None | Anything, at scale, reliably |
| 5. Full headless browser farms | Very high | ✅ Yes | Very high | Massive-scale, adversarial targets |
Let's dig into each.
1. requests + BeautifulSoup: The Classic
For a simple static page, the old faithful still works:
import requests
from bs4 import BeautifulSoup
resp = requests.get("https://example.com")
soup = BeautifulSoup(resp.text, "html.parser")
print(soup.get_text())
The good: No dependencies beyond two libraries, zero cost, full control.
The bad: You are on your own for everything. JavaScript? Not rendered. Layout changed? Your selectors break. Cookie banners, Cloudflare, or bot checks? You are suddenly reverse-engineering HTML for a living.
For a quick one-off on a page you trust, it's fine. For anything at scale, it becomes a part-time job.
2. Playwright / Selenium: The Heavy Lifter
When a page renders everything with JavaScript, you need a real browser:
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto("https://example.com")
print(page.inner_text("body"))
browser.close()
The good: Renders anything a real user sees, can log in, click, and scroll.
The bad: Heavy. Each browser instance eats hundreds of MB of RAM. You have to manage selectors, waits, and timeouts. And the maintenance burden is real — every site redesign means updating your scripts.
Playwright is unbeatable for interacting with a page (clicking buttons, filling forms). But if you only need the content, it's overkill.
3. Trafilatura: The Article Specialist
If you already have the raw HTML and just need the readable text, Trafilatura is a strong pick:
import trafilatura
downloaded = trafilatura.fetch_url("https://example.com")
text = trafilatura.extract(downloaded)
print(text)
The good: Clever heuristics that strip navigation, ads, and boilerplate, leaving the main article body.
The bad: It still can't render JavaScript, and you still fetch the HTML yourself. You also get text — not clean Markdown or structured JSON.
4. A Dedicated Extraction API: The No-Maintenance Option
This is where the Web to Markdown/JSON API comes in. Instead of maintaining a scraper, you make one HTTP call:
curl -X POST https://web2md-api-production-d822.up.railway.app/extract \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "format": "markdown"}'
You get back clean, structured content:
{
"success": true,
"title": "Example Domain",
"content": "# Example Domain\n\nThis domain is for use...",
"description": "",
"author": "",
"published_date": "",
"word_count": 24,
"response_time_ms": 177
}
Three formats are supported via the format field:
-
markdown— clean, human-readable Markdown -
json— a structured JSON representation of the page -
text— plain text with all markup stripped
You can also cap the response size with max_length (up to 50000 characters) to keep payloads snappy.
The good: No scraping infrastructure to maintain. Handles rendering, extraction, and formatting for you. Works from any language with an HTTP client.
The bad: It's a network dependency (though that's true of any API), and the free tier is capped at 50 requests per day.
For most personal projects and MVPs, 50 requests a day is plenty. If you outgrow it, paid plans are available on RapidAPI.
Here's the same call from Python:
import requests
resp = requests.post(
"https://web2md-api-production-d822.up.railway.app/extract",
json={"url": "https://example.com", "format": "markdown", "max_length": 50000},
)
data = resp.json()
print(data["title"])
print(data["content"])
5. Headless Browser Farms: The Nuclear Option
For massive-scale, adversarial targets, some teams run their own fleet of headless browsers behind proxies and CAPTCHA solvers.
The good: Maximum control, works on almost anything.
The bad: You are now running infrastructure. Proxy rotation, retry logic, monitoring, IP reputation management — this is a full engineering project, not a side task.
Unless scraping is your product, this is almost never the right call.
Which Should You Choose?
Here's my honest recommendation:
- Just need the content of a page? → Use the Web to Markdown/JSON API. One call, no maintenance.
- Need to interact with a page (click, log in, fill forms)? → Playwright.
- Already have HTML and want the article body? → Trafilatura.
-
Scraping a static page you fully control? →
requests+ BeautifulSoup. - Scraping at enormous scale? → A browser farm, but budget accordingly.
For the vast majority of developers building reading lists, AI pipelines, or content tools, the API approach wins on time-to-value. You go from "I should build a scraper" to "I have the content" in about five minutes.
Wrapping Up
Web scraping doesn't have to be a maintenance nightmare. The right tool depends on what you actually need — content vs. interaction, scale, and how much infrastructure you want to own.
If you just want clean Markdown or JSON from any URL without the headache, give the Web to Markdown/JSON API a try — the free tier is enough to test the waters.
Have a favorite scraping approach I missed? Drop it in the comments.
Top comments (0)