DEV Community

Survivalengine2
Survivalengine2

Posted on

Building a Resilient, Claude-Powered Web Scraper with YAML (Self-Hosted)

#webdev #ai #python #opensource

The Problem with Traditional Web Scraping

If you've maintained web scrapers at scale, you know the routine: a site updates its layout, class names change, and your XPath or CSS selectors instantly break. You end up spending more time fixing broken selectors than actually using the extracted data.

While cloud-based AI scrapers exist, they usually come with heavy monthly subscriptions and usage caps.

I wanted a solution that was resilient, self-hosted, and cost-effective. That’s why I built HYAE.

What is HYAE?

HYAE is a self-hosted, YAML-driven scraping engine powered by Claude AI. Instead of hardcoding fragile DOM selectors, you define your target output structure in a simple YAML config, and Claude handles the extraction from raw HTML.

Key features include:

  • YAML Configuration: Define clean target schemas without writing complex parsing scripts.
  • Claude-Powered Parsing: Adapts gracefully when underlying site layouts or class names shift.
  • Polite Crawling Defaults: Native support for robots.txt, custom crawl delays, and rate limits to respect target servers.
  • Bring Your Own Key (BYOK): Run it locally or on your own server using your own Anthropic API key. You only pay for the exact tokens you use without recurring platform markups.

How It Works

  1. Define Your Schema: Create a simple YAML file specifying the fields you want to extract (e.g., product titles, prices, specs).
  2. Run the Engine: HYAE fetches the page politely according to defined rate limits.
  3. Structured Output: Claude parses the raw HTML and returns structured JSON or CSV data ready for your pipeline.

Building in Public

HYAE is launching on Product Hunt next Tuesday, October 20th for a one-time £29 license.

I’d love to hear how fellow developers are handling fragile scraping pipelines or if you've tried using LLMs for DOM parsing in your own workflows!

Check out the site here: https://survivalengine2.co.uk

Top comments (0)