DEV Community

Mohamed Senator
Mohamed Senator

Posted on

Turn any web page into typed JSON for your AI agent (with a source quote for every value)

LLM agents are great at reasoning and bad at reading messy HTML. The usual fix is a custom scraper per site, which breaks every time the site changes.

Webpage to JSON for AI Agents takes a URL and a plain-English description of the fields you want, and returns typed JSON. Every value comes with a quote from the page, so you can check it isn't made up.

Input:

{
  "urls": ["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"],
  "instruction": "Extract the book title, price as a number, currency code, and whether it is in stock."
}
Enter fullscreen mode Exit fullscreen mode

Output (real run):

{
  "data": { "title": "A Light in the Attic", "price": 51.77, "currencyCode": "GBP", "inStock": true },
  "evidence": { "title": "# A Light in the Attic", "price": "£51.77" },
  "unverifiedFields": []
}
Enter fullscreen mode Exit fullscreen mode

Useful options

  • JSON schema: pass extractionSchema when you need an exact shape.
  • List mode: set listMode: true to get many records from one page, such as product grids or job boards.
  • Strict evidence: requireEvidence is on by default. A value without a source quote is flagged in unverifiedFields.

Call it from code

from apify_client import ApifyClient
client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("s3nafps/webpage-to-json-for-ai-agents").call(run_input={
    "urls": ["https://example.com/pricing"],
    "instruction": "List each plan with name, monthly price and included seats."
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["data"])
Enter fullscreen mode Exit fullscreen mode

Use it as an MCP tool: add https://mcp.apify.com/?actors=s3nafps/webpage-to-json-for-ai-agents to Claude, Cursor or any MCP client, and your agent can call it directly.

The LLM cost is included. You pay $0.03 per page with data and nothing for pages where no data was found.

Top comments (0)