The latest articles on DEV Community by Survivalengine2 (@survivalengine2_e4d0166ad). https://dev.to/survivalengine2_e4d0166ad https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4186582%2F60c55c4e-de36-4394-abd6-1005a9e4743d.jpg https://dev.to/survivalengine2_e4d0166ad en Survivalengine2 Thu, 15 Oct 2026 10:50:02 +0000 https://dev.to/survivalengine2_e4d0166ad/building-a-resilient-claude-powered-web-scraper-with-yaml-self-hosted-1c2f https://dev.to/survivalengine2_e4d0166ad/building-a-resilient-claude-powered-web-scraper-with-yaml-self-hosted-1c2f <h3> The Problem with Traditional Web Scraping </h3> <p>If you've maintained web scrapers at scale, you know the routine: a site updates its layout, class names change, and your XPath or CSS selectors instantly break. You end up spending more time fixing broken selectors than actually using the extracted data.</p> <p>While cloud-based AI scrapers exist, they usually come with heavy monthly subscriptions and usage caps. </p> <p>I wanted a solution that was <strong>resilient</strong>, <strong>self-hosted</strong>, and <strong>cost-effective</strong>. That’s why I built <strong>HYAE</strong>.</p> <h3> What is HYAE? </h3> <p>HYAE is a self-hosted, YAML-driven scraping engine powered by Claude AI. Instead of hardcoding fragile DOM selectors, you define your target output structure in a simple YAML config, and Claude handles the extraction from raw HTML.</p> <p>Key features include:</p> <ul> <li> <strong>YAML Configuration:</strong> Define clean target schemas without writing complex parsing scripts.</li> <li> <strong>Claude-Powered Parsing:</strong> Adapts gracefully when underlying site layouts or class names shift.</li> <li> <strong>Polite Crawling Defaults:</strong> Native support for <code>robots.txt</code>, custom crawl delays, and rate limits to respect target servers.</li> <li> <strong>Bring Your Own Key (BYOK):</strong> Run it locally or on your own server using your own Anthropic API key. You only pay for the exact tokens you use without recurring platform markups.</li> </ul> <h3> How It Works </h3> <ol> <li> <strong>Define Your Schema:</strong> Create a simple YAML file specifying the fields you want to extract (e.g., product titles, prices, specs).</li> <li> <strong>Run the Engine:</strong> HYAE fetches the page politely according to defined rate limits.</li> <li> <strong>Structured Output:</strong> Claude parses the raw HTML and returns structured JSON or CSV data ready for your pipeline.</li> </ol> <h3> Building in Public </h3> <p>HYAE is launching on <strong>Product Hunt next Tuesday, October 20th</strong> for a one-time £29 license. </p> <p>I’d love to hear how fellow developers are handling fragile scraping pipelines or if you've tried using LLMs for DOM parsing in your own workflows!</p> <p>Check out the site here: <a href="https://survivalengine2.co.uk" rel="noopener noreferrer">https://survivalengine2.co.uk</a></p> ai webdev python opensource