For years, web scraping has been the duct tape behind AI agents trying to interact with websites. It works—until it doesn't. HTML changes, pages lag, and what should be a 100ms query becomes a 4-second screen-parsing mess.
I believe the web needs native infrastructure for agents. Just as robots.txt standardized how crawlers engage with websites, it's time for something similar for AI agents: a declarative, structured protocol.
So I built IntentQL.
🔧 The Problem: Scraping Is a Bottleneck
When an AI agent wants to book a hotel, it doesn't "understand" the page—it scrapes it. That means parsing layout-heavy HTML, guessing at form inputs, and praying nothing broke.
A recent test showed:
Scraping the hotel booking UI took 4.46 seconds
IntentQL took 0.049 seconds
That's a 91x performance improvement. But more importantly, it's stable. HTML can change without breaking the contract.
🌐 What Is IntentQL?
IntentQL is a declarative JSON endpoint websites can serve (like Agent.json) to expose agent-compatible actions. These actions are structured, timestamped, and composable.
An agent can ask: "What can I do here?" — and get a machine-readable list of intents like:
{
"intents": [
{
"name": "book_room",
"description": "Book a hotel room",
"parameters": ["check_in", "check_out"]
}
]
}
No scraping. No guessing. No fragility.
🚀 Live Demo
I built a live side-by-side demo:
https://widget-k3vj-a6qvo466v-lee-manleys-projects.vercel.app/
Left side: full HTML scraping
Right side: direct JSON query using IntentQL
Includes timestamps + graphics
Try it. Embed it. Share it. This is how agents should interact with the web.
🚧 Who Is This For?
Browser teams (Opera, Arc, Perplexity): no more reverse-engineering
Web devs: expose actions natively without API bloat
Standards bodies: time to start the conversation
If you're working on agent-first experiences, this is the missing layer.
👀 What Happens Next?
I'm sharing this across Hacker News, Twitter, and with the W3C AI Agent Protocol community. This isn't a product pitch—it's infrastructure. I'm here to help define the standard.
Spec: https://intentql.dev
Contact: lee@intentql.dev
Want to implement this on your site? There's a one-file guide. Need help? I offer consulting.
Let's give AI agents a better way to use the web.
Built in the open. Maintained at intentql.dev.
Top comments (2)
The biggest blockers are nailing standardized agent identity/consent and the ops tax of maintaining stable intent contracts. Structured endpoints are snappy if you bake in timeouts, Idempotency-Key, retries, ETag/caching, and embedded rate_limits/service_status. Spec-wise, ship JSON Schema, first‑class flows with continuation_id and explicit state machines, clear composability, semver, i18n, standardized result_type, and cursor pagination. Plant the flag at W3C with a neutral foundation, public RFCs and conformance tests. Quick wins are travel/e‑com/ticketing/productivity with ROI from less scraping and higher conversion, plus standard telemetry, change feeds/webhooks, and shadow/percent/A‑B rollouts; the demo’s great-now add standardized errors, auth metadata, and first‑class flows.
Thank you OnlineProxy
Really appreciate you spelling this out — super sharp take. You’re absolutely right: identity, consent, and long-term contract stability are the linchpins here. Agent-powered interfaces aren’t a maybe anymore — they’re the next layer, and it’s on us to shape that foundation. I’m less concerned with polish and more focused on functionality: if agents can interact cleanly and reliably, that’s a win. Your thoughts on flows, standard errors, telemetry, and grounding this at W3C hit exactly where we’re headed. Big thanks for pushing the thinking forward.