Every AI agent project I've worked on eventually needs to shove a web page into LLM context. The pattern is always the same: detect a URL, fetch the HTML, strip the boilerplate, convert to Markdown, feed the model. And every team writes their own version of this — a scraper that works for their sites, breaks on the next one, and rots on a Friday afternoon when the target site changes its DOM.
I've maintained three different versions of this pipeline in the last year alone. I'm done rewriting it.
The scrapers people write (and why they break)
The naive approach is requests + BeautifulSoup:
import requests
from bs4 import BeautifulSoup
resp = requests.get("https://example.com")
soup = BeautifulSoup(resp.text, "html.parser")
# Good luck stripping nav, ads, sidebars, footers...
It works for exactly one site. Every new domain needs a new selector. If the site ships JavaScript-rendered content, you get back a <div> with a loading spinner. So you add Playwright or Puppeteer, and now you're maintaining a headless browser setup too.
The next step is a readability library — Mozilla's Readability, go-readability, or Python's trafilatura / html2text:
# pip install trafilatura
import trafilatura
downloaded = trafilatura.fetch_url("https://example.com")
result = trafilatura.extract(downloaded, output_format="markdown")
Better, but you still handle fetch errors, timeouts, encoding edge cases, and the output quality varies wildly per site. Some code blocks get mangled. Tables disappear. You end up with a configurable pipeline that mostly works but needs a fix every couple of weeks.
Then there are the hosted options — Jina Reader (r.jina.ai/<url>) or Firecrawl. These work, but they're REST round-trips from inside your agent loop: HTTP call out, HTTP call back, parse JSON, hope the schema didn't change. They also mean your agent can't do anything until the network round-trip completes, which in a tight loop adds latency that compounds.
What I actually want
A single command. No HTTP plumbing. No curl. No API key. No pip install of a library I'll need to update. Just: here's a URL, give me Markdown.
# The dream
result = plainweb.fetch("https://docs.example.com/guide")
# result.content == clean Markdown
That's exactly what plainweb is — one of the installable apps in Pilot Protocol's app store. It's a typed local call to a capability that runs on your daemon:
pilotctl appstore call io.pilot.plainweb plainweb.fetch '{"url": "https://example.com"}'
Reply is JSON — { "content_type": "text/markdown", "content": "# Title\\n\\nBody..." }. No HTML. No scraping code. No maintenance.
Discover → Install → Call
The loop is three commands, and you do the discover-and-install once:
# 1. Discover (in case it's new to you)
pilotctl appstore catalogue | grep plainweb # io.pilot.plainweb
# 2. Install once (auto-spawned by the daemon)
pilotctl appstore install io.pilot.plainweb --force
# 3. Call any time
pilotctl appstore call io.pilot.plainweb plainweb.fetch '{"url": "https://dev.to/pstayet"}'
That's it. The daemon supervises the app, re-checks its signature on every spawn, and handles the underlying HTTP and rendering. Your agent never manages a connection pool, never parses HTML, never configures a headless browser.
How it works under the hood
Plainweb uses a cost-aware fetch ladder: for most pages it goes straight to go-readability (fast, no browser), which handles code blocks and tables well. If the page is JS-heavy, it falls back to headless Chrome automatically. The result goes through html-to-markdown v2 for the conversion. You don't pick the ladder — it picks itself based on what the page needs.
This matters because in an agent loop, latency kills. A readability-only fetch takes milliseconds. A full Chrome render takes seconds. Plainweb optimizes for the common case transparently.
Practical: feed documentation into your agent
Here's a concrete use case: your agent needs to research a library's docs before writing code. Instead of hardcoding a scraper per docs site, you call plainweb directly from your agent's toolchain:
from hermes_tools import terminal
# The one command
result = terminal("pilotctl appstore call io.pilot.plainweb plainweb.fetch '{\"url\": \"https://docs.pilotprotocol.network\"}'")
# result contains clean Markdown — feed it straight into context
No pip install, no requirements.txt entry, no scraper to maintain. The app is signature-verified and auto-supervised by the daemon — supply-chain safe.
Why this beats every DIY approach
| Approach | Maintenance | Latency | Auth needed |
|---|---|---|---|
| requests + BeautifulSoup | Per-site selectors rot | Low | No |
| Playwright/Puppeteer | Browser deps + config | High | No |
| Readability library | Version upgrades + edge cases | Low | No |
| Jina Reader / Firecrawl | REST schema, rate limits, API keys | Medium | Key needed |
| Plainweb via Pilot | None (app maintained) | Low (adaptive) | None |
The key difference: a local typed call vs an API call. Plainweb runs on your daemon's process — no HTTP in/out, no JSON parsing of a REST response, no rate-limit errors. The daemon handles all of that.
The bigger pattern
plainweb is one app. The Pilot app store has dozens more — web search (cosift), people/company intelligence (sixtyfour), container management (docker), runtime security (AEGIS), disposable microVMs (smolmachines). The pattern is the same for all of them: discover → install → call.
For an agent, this turns "maintaining infrastructure" into "installing an app." The difference between writing a scraper and calling plainweb is the difference between building your own database and installing PostgreSQL. One is productive work. The other is a tax you shouldn't be paying.
Get started:
curl -fsSL https://pilotprotocol.network/install.sh | sh
pilotctl appstore install io.pilot.plainweb --force
pilotctl appstore call io.pilot.plainweb plainweb.fetch '{"url": "https://example.com"}'
Top comments (0)