DEV Community

Mohamed Senator
Mohamed Senator

Posted on Edited on

Turn any web page into typed JSON for your AI agent (with a source quote for every value)

Webpage to JSON for AI Agents demo
Watch the full video

LLM agents are great at reasoning and bad at reading messy HTML. The usual fix is a custom scraper per site, which breaks every time the site changes.

Webpage to JSON for AI Agents takes a URL and a plain-English description of the fields you want, and returns typed JSON. Every value comes with a quote from the page, so you can check it isn't made up.

Input:

{
  "urls": ["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"],
  "instruction": "Extract the book title, price as a number, currency code, and whether it is in stock."
}
Enter fullscreen mode Exit fullscreen mode

Output (real run):

{
  "data": { "title": "A Light in the Attic", "price": 51.77, "currencyCode": "GBP", "inStock": true },
  "evidence": { "title": "# A Light in the Attic", "price": "£51.77" },
  "unverifiedFields": []
}
Enter fullscreen mode Exit fullscreen mode

Useful options

  • JSON schema: pass extractionSchema when you need an exact shape.
  • List mode: set listMode: true to get many records from one page, such as product grids or job boards.
  • Strict evidence: requireEvidence is on by default. A value without a source quote is flagged in unverifiedFields.

Call it from code

from apify_client import ApifyClient
client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("s3nafps/webpage-to-json-for-ai-agents").call(run_input={
    "urls": ["https://example.com/pricing"],
    "instruction": "List each plan with name, monthly price and included seats."
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["data"])
Enter fullscreen mode Exit fullscreen mode

Use it as an MCP tool: add https://mcp.apify.com/?actors=s3nafps/webpage-to-json-for-ai-agents to Claude, Cursor or any MCP client, and your agent can call it directly.

The LLM cost is included. You pay $0.03 per page with data and nothing for pages where no data was found.

Top comments (0)